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Preface 


In  August  1985,  the  Rome  Air  Development  Center  selected  Sanders  Associates,  Inc.  to  evaluate 
the  software  development  process  for  Artificial  Intelligence  (Al)  systems  and  postulate  a  software 
acquisition  model.  To  accomplish  these  objectives,  Sanders  devised  a  strategy  consisting  of  the 
following  major  elements: 

•  Literature  review; 

•  Case  study  analysis;  and 

•  Consultation  with  experienced  AI  system  developers. 

The  case  study  analyses  represent  historical  data  on  26  knowledge  base  system  (KBS)  development 
efforts.  Because  the  case  data  focuses  on  KBS  software,  the  acquisition  model  developed  pertains 
to  KBS  efforts. 

The  results  of  this  study  are  presented  in  a  two  volume  report.  Volume  1  presents  observations 
made  during  the  analysis  of  KBS  software  developments  as  well  as  summaries  of  the  case  study 
data.  A  comparison  of  the  KBS  development  process  to  DOD-STD-2167  is  also  made. 

Volume  II  presents  a  KBS  process  model  as  well  as  a  postulated  customer/developer  interface 
model.  A  comparison  of  the  postulated  model  with  DOD-STD-2167  and  DOD-STD-2167  A  (draft) 
is  made  in  terms  of  activities,  products,  reviews  and  baselines. 


SECTION  1 


Introduction 


1.1  Problem  Definition 

To  date,  many  Artificial  Intelligence  (AI)  systems  have  been  developed  in  university  and  other 
research  environments  with  relatively  few  oriented  towards  Department  of  Defense(DOD)  applica¬ 
tions.  The  scale  and  complexity  of  these  existing  systems  is  substantially  less  than  that  anticipated 
for  the  Strategic  Defense  Initiative  (SDI)  Battle  Management/Command  Control  and  Communica¬ 
tions  ( BM/C ')  Technology  Program.  In  addition,  the  manner  in  which  these  Al  systems  have  been 
developed  is  largely  foreign  to  the  DOD  software  acquisition  process  (e  g  DOD-STD-2167)  .  For 
instance,  conventional  software  issues  such  as  design  reviews,  programming  languages,  documenta¬ 
tion,  quality  assurance  procedures,  testing  and  other  methods  commonly  used  in  military  system 
acquisition  are  nonexistent,  inappropriate  or  radically  different  in  the  Al  software  development 
process.  Because  DOD  envisions  a  need  for  A I  system  technology  to  address  many  components  of 
the  SDI,  a  clearer  understanding  of  these  differences  is  required.  Namely,  new  policies,  procedures, 
standards  and  contracting  mechanisms  must  be  in  place  to  ensure  a  high  success  rate  in  acquiring 
AI  systems  in  general.  Consequently,  specific  stated  areas  of  interest  to  DOD  are: 

•  Analysis  of  the  AI  software  development  process  to  determine  unique  characteristics  and  needs; 

•  Development  of  a  model(s)  for  AI  software  acquisition  that  satisfies  both  military  needs  and 
unique  AI  requirements.  The  model  should  at  least  address  the  following  areas: 

-  documentation  standards; 

-  review  procedures; 

--  rigorous  testing  methods;  and 

-  contract  mechanisms. 

•  Analysis  and  specification  for  interfacing/integrating  Al  and  conventional  software  languages 
and  processes. 

The  Rome  Air  Development  Center  (RADC),  Air  Force  Systems  Command,  at  C.rifTiss  Air  Force 
Base,  N.Y.,  was  tasked  by  DOD  to  respond  to  SDI  needs  concerning  technology  critical  to  I1M  C' 
Having  solicited  public  response  in  the  form  of  proposals,  RADC  selected  Sanders  Associates,  Inc 
to  perform  and  report  on  a  specific  study  consistent  with  DOD  interests  as  stated  above. 


1.2  Solution  Strategy 

To  address  the  DOD  needs  as  delineated  in  the  previous  section,  Sanders  proposed  a  two  phase 
approach.  Phase  I  focused  on  an  analysis  of  the  AI  software  development  process  The  analysis 


1.2  Solution  Strategy 


consisted  of  collecting  case  study  data  for  AI  systems  which  had  either  completed  development  or 
were  well  underway  towards  completing  a  prototype.  During  the  Phase  I  data  collection  activity, 
it  was  learned  that  case  study  data  was  only  readily  available  for  Knowledge  Based  Systems  (KBS) 
and  not  generally  available  for  other  Al  application  areas  (e.g.  signal  processing,  natural  language 
processing,  etc.).  Because  of  this,  the  scope  of  the  study  concentrated  on  KBS.  Phase  I!  focused 
on  the  development  of  a  KBS  software  acquisition  model.  Issues  germane  to  KBS  and  conventional 
software  interfaces/integration  are  considered  to  be  a  part  of  both  phases. 

In  Phase  I,  the  analysis  effort  included  an  extensive  literature  search/review,  detailed  case  study 
evaluations  and  an  assessment  of  DOD-STD-2167.  The  basis  of  the  adopted  strategy  was: 

•  Extract  from  the  published  literature  as  much  information  as  possible  concerning  KBS  software 
development  techniques  and  characteristics; 

•  Design  a  comprehensive  questionnaire  to  extract  data  from  experienced  KBS  software  builders 
on  specific  systems; 

•  Consult  in-house  expert  system  builders  to  enhance  the  understanding  of  the  material  re¬ 
viewed; 

•  Study  DOD-STD-2167  methodology  for  use  as  the  basis  for  defining  deviations  appropriate 
for  KBS  software  development. 

This  approach  enabled  the  study  team  to  meet  the  requirements  of  Phase  I.  Specifically,  KBS 
software  development  characteristics  were  defined  and  deviations  from  the  conventional  software 
development  process  were  identified. 

The  Phase  I  approach  also  resulted  in  the  collection  of  data  pertinent  to  the  Phase  II  effort.  Namely, 
case  study  analyses,  particularly  in  the  area  of  expert  systems,  led  to  the  identification  of  KBS  areas 
critical  to  the  SDI  effort.  Case  study  participants  also  provided  substantial  information  concerning 
“lessons  learned”  which  proved  to  be  invaluable  in  terms  of  defining  a  standard  approach  that 
avoids  the  pitfalls  already  experienced  by  others. 

Specific  goals  for  Phase  II,  the  final  phase  of  the  software  project,  were  to  define  a  model  that 
specified  acquisition  and  development  approaches  for  Knowledge  Based  System  applications.  The 
models  developed  define  the  activities,  products,  reviews  and  baselines  for  the  development  of  KBS 
software.  They  also  identify  management  needs  for  visibility  and  control  over  developing  products 
as  well  as  the  required  delivery  of  quality  products  within  cost  and  schedule. 

Additional  activities  designed  to  satisfy  the  goals  of  Phase  II  included  a  strong  coupling  with  the 
DOD-STD-2167  Revision  A  activities,  gaining  insight  into  software  quality  metrics  research  and 
development  and  continued  data  collection  as  new  data  sources  were  defined.  The  coupling  with 
the  2167  Revision  A  activities  was  necessary  to  produce  a  model  which  is  compatable  with  and 
interfaces  to  the  world  of  conventional  software  development. 

The  results  of  the  Phase  II  modeling  effort  are  documented  in  Volume  II  of  this  final  report.  The 
remainder  of  this  report  is  outlined  as  follows: 


Section  2  contains  a  discussion  of  the  methods  used  to  gather  all  available  data  concerning 
the  KBS  software  development  process. 


1.2  Solution  Strategy 


•  Observations  made  through  analysis  of  the  data  obtained  aie  presented  in  Section  3  These 
observations  pertain  to  both  the  conventional  and  KBS  software  approaches,  including  the 
difficulties  inherent  to  both  methods. 

•  A  detailed  evaluation  of  the  case  study  data  obtained  is  contained  in  Section  1  Common 
aspects  and  important  features  are  highlighted. 

•  Section  5  presents  a  synopsis  of  KBS  development  methods  versus  DOD-STD-2167,  conven¬ 
tional  and  KBS  software  interface  issues  as  well  as  the  application  of  KBS  to  SD1  requirements 

•  A  list  of  references,  bibliography,  glossary,  and  a  list  of  acronyms  follow  Section  5 

•  Lastly,  the  report  includes  several  appendices: 

-  Appendix  A  details  problems  cited  in  the  development  of  KBS  and  conventional  software; 

-  Appendix  B  presents  the  case  study  questionnaire;  and 

-  Appendix  C  presents  summaries  of  the  case  study  responses 
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SECTION  2 


2.1  Literature  Search 


The  data  gathering  effort  began  with  a  comprehensive  literature  search  extending  to  both  on-line 
and  hardcopy  sources  of  information.  The  computerized  DIALOG  system1  was  used  to  access  the 
following  databases: 


1NSPEC  (Information  Services  in  Physics,  Electrotechnology,  Computers  and  Control) 


•  ABI/Iiiform  (American  Business  Information) 


The  Computer  Database 


NTIS  (National  Technical  Information  Service) 


The  DROLS  (Defense  R1)T  k  E  On-Line  System)  search  service  was  also  used  to  access  the  DTK ' 
(Defense  Technical  Information  Center)  database. 


Keywords  pertinent  to  the  study  were  used  to  identify  articles  from  the  various  dat  abases  for  furt  her 
inspection.  Numerous  articles  covering  a  variety  of  relevant  topics  such  as  artificial  intelligence, 
expert  systems,  natural  language  processing  and  battle  management  were  ordered  as  a  result  of  the 
on-line  search  activity. 


In  addition  to  computerized  search  techniques,  many  hardcopy  sources  of  information  were  reviewed 
such  as  textbooks,  conference  proceedings  and  journals  A  complete  annotation  of  all  resources  used 
in  the  preparation  of  this  document  is  presented  in  the  List  of  References  and  the  Bibliography 


Tin’  literature  search  was  an  ongoing  process  since  its  inception  Namely,  particular  rcferciu  es 
listed  at  the  back  of  an  article  or  textbook  under  review  were  often  ordered  This  ripple  elleci 
enabled  the  study  team  to  obtain  and  review  an  exhaustive  set  of  resources  corresponding  to  the 
subject  study 


The  other  medium  that  was  used  for  obtaining  references  and  information  about  Al  development 
issues  was  the  AHPA/USENET  groups  for  Al  including  both  the  moderated  and  unmoderated 
newsgroups  Numerous  articles,  technical  reports  and  conference  proceedings  were  ordered  and 
reviewed  as  a  result  of  references  from  the  network  discussions. 


DIALOG  Information  Services.  Inc  ,  n  wholly  owned  subsidiary  of  Lockheed  imhi 
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2.2  Case  Studies 


2.2.1  Questionnaire 


A  comprehensive  questionnaire  was  designed  and  used  as  the  basis  for  conducting  the  AI/KBS 
system  case  studies.  The  questionnaire,  contained  in  Appendix  B,  is  divided  into  four  parts: 


1.  Introduction 


2.  Background 


3.  Development  Cycle,  and 


4.  Miscellaneous. 


The  Introduction  put  the  questionnaire  in  perspective  by  discussing  the  concept  of  a  software  de¬ 
velopment  model.  The  Background  section  solicited  general  information  about  the  subject  system, 
such  as  purpose,  the  number  of  experienced  engineers  assigned  to  the  project  and  the  extent  of 
success  attributable  to  the  system.  The  Development  Cycle  section  contained  the  most  questions 
and  was  subdivided  into  the  following  categories: 


•  General  Procedures 


Requirements  Definition/ Analysis 


System  Construction 


System  Evaluation/Validation,  and 


•  Field  Support. 


The  overall  purpose  of  the  Development  Cycle  section  was  to  obtain  detailed  technical  and  man¬ 
agerial  information  concerning  the  scope,  development  and  acceptance  of  the  system. 


Lastly,  the  section  titled  Miscellaneous  was  used  to  capture  data  on  lessons  learned  as  well  as  allow 
the  respondent  to  include  pertinent  information  that  had  not  already  been  requested. 


Most  of  the  responses  received  were  in  written  form.  In  a  few  cases,  oral  responses  were  obtained 
using  an  interview  or  discussion  forum  with  the  respondent.  In  these  cases,  a  written  annotation 
of  the  discussion  was  generated  and  sent  to  the  respondent  to  verify  the  accuracy  of  the  reported 
information. 


2.2.2  Selection  Criteria  for  Cases  Studied 


Once  the  questionnaire  was  finalized,  the  primary  goal  was  to  obtain  as  many  responses  as  possible. 
Maximizing  the  number  of  samples  available  for  analysis  enabled  conclusions  to  be  drawn  which 
more  closely  reflected  the  general  AI/KBS  community.  Distribution  of  the  questionnaire  was  also 
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2.2.3  List  of  Participants 


made  to  a  wide  variety  of  companies.  In  this  wa  ,  data  was  captured  pertaining  to  ditlerenl 
application  areas  and  procedural  methods  for  developing  Al/KBS  software. 

The  case  selection  process  began  by  identifying  two  in-house  KBS  efforts  From  that  point,  the 
literature  search  and  review  uncovered  many  companies  and  universities  that  are  active  in  KBS 
software  development .  Because  most  of  the  university  efforts  are  experimental  in  nat  ure  and  without 
strict  schedules  for  completing  defined  milestones,  the  tendency  was  to  concentrate  on  the  industrial 
sector.  Companies  that  claimed  to  have  built  successful  A1  systems  were  of  the  most  interest 

In  addition  to  the  literature  search,  data  was  obtained  on-line  from  the  Commerce  Business  Daily 
(CBD)  files  concerning  AI  awards  since  September  1982.  Based  on  the  descriptions  of  the  projects 
awarded,  candidates  whose  work  seemed  most  pertinent  to  the  subject  study  were  selected  In 
addition,  discussions  were  held  with  cognizant  government  agencies  to  ensure  appropriateness  of 
effort  and  verify  claims  in  the  literature. 

As  mentioned  in  the  following  section,  attendance  at  various  workshops/  seminars  resulted  in  many 
contacts  who  were  interested  in  responding  to  our  questionnaire.  Lastly,  some  of  the  contacts  made 
resulted  in  references  to  other  sources  active  in  the  KBS  arena 

2.2.3  List  of  Participants 

Based  on  the  selection  process,  over  50  companies/agencies  were  contacted  Initial  contacts  were 
generally  made  by  telephone  to  introduce  the  study  and  determine  whether  or  not  the  source 
was  interested  in  participating.  In  the  majority  of  cases,  the  response  to  participate  was  positive 
Although  more  than  40  questionnaires  were  sent  out  to  interested  parties,  many  respondents  wen- 
prohibited  from  participating  on  the  basis  of  proprietary  information  Consequently,  2b  completed 
questionnaire  responses  were  received.  The  list  of  respondents  is  shown  in  Table  2.2  3  I  along 
with  the  name  of  the  system  to  which  the  response  applies.  Case  analyses  are  presented  in  Section 
4  and  summaries  corresponding  to  the  questionnaire  responses  for  each  participant  art  shown  m 
Appendix  C. 


2.3  Other  Methods 
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In  addition  to  the  literature  search  and  case  studies,  project  members  at  tended  several  Ai  seminars 
to  obtain  information  concerning  the  subject  study.  Individuals  experienced  in  AI  software  devel¬ 
opment  were  consulted  as  needed  to  enhance  the  team’s  understanding  of  the  discipline  Lastly, 
to  gather  data  pertinent  to  the  ongoing  SDI  architecture  studies,  a  trip  was  made  to  the  Sl)l  Li¬ 
brary  in  Falls  Church,  Virginia.  These  information  gathering  methods  are  described  further  in  the 
succeeding  paragraphs 

2.3.1  EIA  Workshop 

During  the  week  of  16  through  20  September  1985,  a  team  representative  was  present  at  an  1  let - 
trouic  Industries  Association  (EIA)  workshop  entitled  “Technology  for  the  1990  's  ’’  This  team 
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2.3.1  El  A  Workshop 


Table  2. 2. 3-1:  List  of  Questionnaire  Respondents 


NAME 


ARINC 

BOEING 


SYSTEM 


STAMP 


BOEING 

BRATTLE  RESEARCH 
CARNEGIE  GROUP 


FREY  ASSOCIATES 
GTE  DATA  SERVICES 
IBM  FSG 
INFERENCE 
INFERENCE 


MITRE  (BEDFORD) 
MITRE  (McLEAN) 
NORTHROP 
PGSC 
PGSC 
PGSC 
SANDERS 
SCHLUMBERGER 
SA<VE 
SAAiE 


Strategic  Force  Management 
Decision  Aid 


Unnamed 

DISPATCHER 

XCON 

PEGASYS 

THEMIS 

COMPASS 

FDRS 

Authorizer’s  Assistant 
Medchec 


Frequency  Hopper  Signal  Identifier 
Pilot’s  Associate 
Liquid  Oxygen  Expert  System 
ANALYST 
ESTAS 
CBTAO 
DART 
SPEA 
TESS 

Dipmeter  Advisor  System 
DSDS 
SFAS 

PRODUCTION  SCHEDULER 


APPLICATION 


Equipment  testability  shell 
Replanning  decision  aid 

Automatic  Target  Recognition 
Text  interpretation 
Factory  monitor/control 
Computer  system  configuration 
Automatic  pagination 
Natural  language  processor 
Fault  diagnosis 
Fault  diagnosis 
Charge  authorizations 
Fraud  detection 
Software  costing 
Signal  identification 
Combat  avionics  assistant 
Fault  diagnosis/detection 
Tactical  planning  system 
Avionics  decision  aid 
Tactical  cost  decision  aid 
Target  decision  aid 
Battle  situation  projections 
Test  equipment  assistant 
Interpretation  of  logs 
Decision  support  tool 
Financial  analysis  system 
Manufacturing  production  scheduler 


invi 


2.3.2  Export  System  Conference 


representative  attended  a  tutorial  on  DOD  Standard  2167  (hereafter  referenced  as  2167)  and  par¬ 
ticipated  in  a  panel  forum  on  Al  and  its  Impart  on  Current  .:><<// irn re  [: nyum  ring  1'iv.rtut  s.  In 
particular,  the  panel  concentrated  on  the  Al  software  development  process  and  2167  Many  features 
of  the  Al  development  process  were  identified,  most  of  which  are  neither  addressed  nor  compal  ible 
with  the  most  recent  version  of  2167,  Specific  areas  of  concern  are  discussed  in  5  1  of  this  report 

In  addition  to  the  information  obtained  at  the  workshop,  attendance  was  valuable  because  it  led 
to  several  contacts  who  responded  to  the  case  study  questionnaire 

2.3.2  Expert  System  Conference 

In  October  1985,  a  team  member  attended  a  two  day  Expert  Systt  ms  C'onft  rt  net  sponsored  by  the 
Data  Processing  Management  Association  (PPMA).  There  were  nineteen  speakers,  three  of  whom 
were  military  personnel  involved  in  Al  applications  work.  Although  the  majority  of  the  speakers 
were  from  the  industrial  sector,  several  were  actually  working  on  Al  systems  for  the  military. 
Pertinent,  topics  covered  included: 

•  managing  the  development  of  large  expert  systems; 

•  software  engineering  methods  for  expert  system  development; 

•  knowledge  acquisition  techniques; 

•  developing  expert  systems  in  AdaJ;  and 

•  a  variety  of  expert  system  applications  and  potential  uses 

The  conference  proved  to  be  valuable  in  terms  of  obtaining  information  germane  to  the  subject 
study  and  identifying  potential  respondents  to  our  case  study  questionnaire 

2.3.3  TI  Satellite  Symposiums 

Team  members  attended  three  Artificial  Intelligence  Satellite  Symposiums  sponsored  by  Texas 
Instruments,  Inc.  (TI).  The  dates  and  titles  are  as  follows 

•  November  l.’l,  1985  l\  noir/t  <l<p  ■  Hast  d  Si/stt  nis  Ami  Jhtir  Applications 

•  June  25,  1986  knowlcdyt  -Hast  d  Systems:  ,4  Step- Hy-.'slt  p  (iindt  In  < . <  thm/  Start*  d 

•  April  8,  1987  An  A  I  Productivity  lioundtable 

Kach  symposium  featured  renowned  speakers  in  the  Al  field  For  example.  Dr  l.dward  l  eigeubaum 
(Stanford  University)  participated  in  all  three  sessions 

lOach  of  those  symposiums  included  descript  ions  of  several  expert  system.-,  in  routine  use.  and  live 
question  and  answer  sessions.  Resource  materials  were  also  provided  including  a  glossary  of  terms 
and  an  extensive  bibliography 
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2.3.4  Export  System  Technology  Transfer  Seminar 


From  May  12-1 -I.  1986.  a  team  member  attended  an  expert  system  seminar  sponsored  by  Digital 
Equipment  Corporation  (DEC).  Practical  strategics  for  managing  expert  system  development  were 
covered  from  three  perspectives:  strategic/business,  technical  and  human  resource/ organizational 
The  faculty  included  DEC  personnel  currently  involved  in  expert  system  development  projects. 
One  of  the  more  valuable  facets  of  the  seminar  was  a  video  taped  case  study,  extending  over  the 
three  day  period.  After  each  of  the  nine  tapes  were  played,  the  attendees  separated  into  groups  to 
analyze  salient  issues  and  recommend  a  course  of  action. 


The  materials  from  the  seminar  include  the  Guti/e  l<>  Expert  Systems  Management  manual  which 
presents  in  detail  DEC's  ten  step  management  procedure. 


2.3.5  Consultation 


Three  individuals  have  been  used  as  consulting  resources  since  the  inception  of  the  project:  Dr. 
Charles  Rich  of  MIT,  Mr.  J.T.  Cinn  and  Mr.  David  Harris,  both  of  Sanders  Associates. 

Dr  Rich,  an  Al  research  scientist  at  MIT,  has  provided  guidance  concerning  our  solution  strategies 
and  reviewed  selected  materials  generated  by  the  project  team.  Dr.  Rich  is  active  in  the  field, 
having  published  numerous  articles,  given  many  Al  related  lectures  and  having  received  several 
grants  from  the  National  Science  Foundation  (NSF)  and  the  Defense  Advanced  Research  Projects 
Agency  (DARPA). 

Mr.  Cinn  and  Mr.  Harris,  both  active  in  the  development  of  Al  software  at  Sanders  Associates, 
have  provided  continuous  support  in  suggesting  case  study  contacts,  answering  questions  concerning 
the  A I  development  process,  reviewing  all  project  documentation  and  making  recommendations  as 
appropriate 


2.3.0  Research  at  SDI  Library 


Two  team  members  visited  the  SDI  library  in  February  1986  to  review  SDI  architecture  study  data 
as  well  as  other  materials  pertinent  to  the  study.  Although  architecture  study  information  was 
unavailable,  several  articles  pertaining  to  the  study  in  general  were  identified  and  ordered. 


2.3.7  Coupling  with  DOD-STD-2 J67 


On  two  different  occasions,  team  members  met  with  the  Joint  Logistics  Commanders  (JLC)  agen¬ 
cies  concerning  the  Logicon,  Inc.  Revision  A  activities  on  2167.  The  intent  of  this  coupling  was  to 
provide  input  to  Revision  A  concerning  Al  system  developments  and  their  applicability  to  the  stan¬ 
dard  Since  the  Revision  A  draft  publication  was  completed  prior  to  the  KBS  modeling  activity,  any 
influence  on  the  proposed  changes  to  2167  was  minimal.  It  is  expected,  however,  that  the  planned 
release  of  Revision  R  will  be  a  more  appropriate  time  to  closely  couple  KBS  system  acquisition  with 
the  development  model  of  2167  and  to  identify  mismatches  requiring  resolution.  Because  of  the 
nature  of  KBS  versus  conventional  software  development  and  a  foreseen  requirement  to  integrate 
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2.3.7  (  onjiling  with  DOD-STD-2107 


both  into  the  same  system  (eg.,  knowledge-based  signal  processing ) .  it  is  important  I  li.n  (iniiiiiiiniis 
dialogue  be  maintained  between  KBS  modeling  activities  and  changes  to  the  conventional  software 
development  models  defined  in  2167 


SECTION  3 


Observations 


3.1  Conventional  Software  Approach 

3.1.1  Standard  Waterfall  Approach  to  Conventional  SW  Development 

Large  scale  DOD  computerized  system  developments,  which  began  in  earnest  in  the  1900’s,  took 
various  approaches  to  the  development  of  software.  Although  there  were  no  formally  defined  models 
for  software  development  within  DOD,  there  evolved  an  accepted  approach  that  became  the  pat¬ 
tern  for  software  developments  acquired  under  contract  to  the  DOD.  This  approach  documented 
by  [6,  Boehm],  (42,  Metzger],  and  others  was  characterized  as  the  waterfall  model  for  software 
development.  Although  the  terminology  and  the  break  points  between  phases  varied,  the  models 
essentially  included  the  phases  of: 


•  requirements  definition; 

•  design, 

•  coding  and  debug; 

•  integration  and  testing;  and 

•  operations  and  maintenance. 


The  Military  Services,  in  the  mid  and  late  197()’s,  began  to  recognize  the  waterfall  model  as  a  in-oper 
representation  of  a  sound  management  approach  and  took  actions  to  embody  this  model  within  (In 
various  policies  and  standards  being  developed  at  that  point  in  time.  For  example,  both  Air  Force 
Regulation  (AFR)  800-14,  written  in  1975,  and  MlL-STD-1679(Navy),  written  in  1978  in  ludt  d 
some  level  of  representation  of  the  waterfall  model.  The  1079  standard  went  even  furl  In  r  and 
included  the  engineering  disciplines  that  were  an  outgrowth  of  the  struct  tired  engineering  revo  ulion 
The  concepts  of  top-down  design  and  structured  programming  were  included  its  rcquircme  its  for 
software  being  developed  for  the  Navy. 

Policies  and  procedures  were  developed  to  address  the  management  and  engineering  problems 
known  to  exist  during  t  he  mid-70’s.  As  the  applical  ion  of  computers  w  it  bin  I  )0| )  syst  em>  expanded . 
problems  with  software  continued  to  occur  and  the  search  to  find  ways  to  improte  the  software 
development  process  was  an  ongoing  one 

The  next,  step  within  DOD  in  the  policy  and  software  development  standards  arena  came  about 
as  a  result,  of  a  workshop  sponsored  by  the  Joint  Logistics  Commanders  Computer  Resources 
Management.  Croup.  The  workshop,  held  in  March  1979,  put  forth  the  recommendation  that  the 
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3.1.2  Reported  Problems  in  Conventional  Software  Development 


Services  should  and  could  pursue  the  development  of  software  under  a  single  policy  and  standard 
and  that  efficiencies  could  be  achieved  with  a  common  approach  within  the  1)01)  community. 

As  a  result  of  the  workshop,  a  working  group  was  formed  to  develop  a  joint  Service  policy  and 
standard  for  the  development  of  software.  The  model  chosen  was  a  slight  modification  to  previously 
mentioned  waterfall  models  and  was  highlighted  by  its  concentration  on  activities,  products,  reviews 
and  baselines  associated  with  the  software  development  process.  Both  the  policy  and  the  standard, 
now  known  as  DOD-STD-2167  and  dated  5  June  1985,  were  developed  in  an  effort  to  improve 
the  DOD  software  development  process  with  an  eye  towards  eliminating  the  types  of  problems 
identified  below.  In  addition  to  the  standard,  a  set  of  data  item  descriptions  (DIDs)  and  changes 
to  Military  Standards  483,  1521,  and  490  were  developed.  This  standards  package  represents  the 
DOD  approach  to  software  development  for  the  1980's. 

3.1.2  Reported  Problems  in  Conventional  Software  Development 

In  preparation  for  the  KBS  modeling  effort,  it  was  necessary  to  review  past  problems  in  conven¬ 
tional  software  development  so  that  past  mistakes  would  not  be  repeated.  Numerous  studies  had 
been  conducted  and  significant  effort  expended  to  define  and  categorize  these  problems.  With  the 
knowledge  of  past  development  problems  in  hand,  the  modeling  effort  could  be  conducted  in  such  a 
manner  that  the  resulting  KBS  development  model  orientation  would  not  repeat  the  past  problems 
of  the  conventional  software  development  world. 

Many  problems  associated  with  conventional  software  surfaced  during  the  development  of  DOD 
Defense  Systems  and  can  be  categorized  best  under  the  following  headings: 

•  Software  Life  Cycle; 

•  Software  Environment; 

•  Software  Product;  and 

•  Software  People. 

The  following  sections  examine  the  major  problems  in  each  of  the  stipulated  categories.  The 
problem  categorization  identified  is  based  on  a  report  issued  by  the  DOD  Joint  Services  Task 
Force,  R>  part  of  the  DOD  Task  Force  on  Software  Problems.  Appendix  A  presents  a  list  of  specific 
conventional  and  Al  software  development  problems.  (17,  DrufTel] 

3. 1.2.1  Software  Life  Cycle 

Software  Life  Cycle  refers  to  the  development  of  conventional  software  from  requirements  defi¬ 
nition  and  analysis  to  system  maintenance  and  quality  assurance.  Software  problems  have  been 
experienced  in  the  following  areas: 

•  Requirements; 

•  Management; 


3.1.2  Reported  Problems  in  Conventional  Software  Development 


•  Acquisition; 

•  Product  Assurance;  and 

•  Transition. 

Requirements  is  the  process  that  involves  the  analysis  and  definition  of  a  system.  In  conventional 
software,  some  of  the  major  difficulties  with  requirements  surface  from  incomplete  or  inaccurate 
requirements  documents  and/or  poor  communication  between  users  and  engineers.  In  general, 
each  of  the  stipulated  problems  can  result  in  increased  costs  and  unacceptable  schedule  delays  A 
system’s  success  is  particularly  affected  when  the  requirements  are  based  primarily  on  available 
time  and  money. 

Difficulties  with  management  of  conventional  software  development  occur  because  of  a  variety  of 
obstacles  that  relate  to: 

•  insufficient  budget  allocation; 

•  unskilled  management; 

•  lack  of  metrics,  models  and  tools; 

•  undefined  software  acquisition  methods;  and 

•  determining  how  to  develop  software  versus  hardware. 

The  ripple  effect  of  unskilled  management  is  reflected  throughout  all  phases  of  the  software  life 
cycle.  For  example,  proper  design  depends  highly  on  complete  requirements  which  in  turn  rely 
heavily  on  good  management  communication.  Another  obstacle  to  good  management  results  from 
budget  estimates  based  on  vague  requirements  or  budgets  determined  from  inaccurate  models  and 
metrics  designed  to  determine  software  costs.  Finally,  without  the  necessary  models,  metrics  and 
tools,  management  does  not  have  the  appropriate  tools  to  adhere  to  software  development  standards 
requiring  status  reporting. 

Acquisition  of  software  and  tools  is  not  fully  defined  for  conventional  software  development  because 
government  software  acquisition  has  historically  been  based  on  the  purchase  of  associated  hardware 
In  the  area  of  tools  application,  configurations  of  software  are  often  not  properly  managed  even 
t  hough  the  tools  exist  to  do  so.  'Pools  are  often  developed  in-house  and  not  made  available  to  ot  hers 
for  their  use. 

Testing  is  not  provided  at  each  life  cycle  phase  because  of  insufficient  funding  and  scheduling 
In  fact,  uncertainties  arise  in  determining  the  amount  and  kind  of  testing  suitable  to  each  life 
cycle  phase.  Testing  problems  occur  most  frequently  where  requirements  are  vague,  incomplete  or 
difficult  to  measure. 

The  transition  of  software,  whether  from  exploratory  research  to  engineering  development,  or  from 
development  to  maintenance,  introduces  numerous  problems  that  adversely  affect  conventional 
software.  Two  transition  examples  from  which  problems  sui faced  are  the  introduction  of  micro¬ 
processors  and  firmware  into  software  development  and  the  distribution  of  management  control 
(a  need  for  a  standard  or  policy  has  arisen';  and  the  transition  of  software  from  research  and 
development  to  operational  systems  (how  should  it  be  accomplished’’) 
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3.1.2  Reported  Problems  in  Conventional  Software  Development 


3. 1.2. 2  Environment 

The  Environment  category  deals  with  the  tools  and  methodologies  used  in  the  development  and 
support  of  computer  software.  Problems  exist  in  the  following  areas: 

•  Disciplined  Methods; 

•  Labor-intensive  activities; 

•  Tools; 

•  Reinvent  ion;  and 

•  Capital  Investment. 

Conventional  software  needs  to  improve  the  use  of  engineering  disciplines.  A  set  of  activities  are 
required  to  develop  and  support  software  throughout  the  entire  life  cycle  in  order  to  create  high 
quality  software  Large  development  efforts  are  performed  by  many  groups,  either  colocated  or 
independently  located.  This  demands  disciplined  control  mechanisms  if  software  development  is  to 
be  properly  managed. 

As  a  labor-intensive  technology,  conventional  software  needs  to  concentrate  on  automating  the 
manual  process  in  order  to  improve  the  efficiency  of  people.  For  example,  the  manual  process  of 
reporting  documents,  status,  etc.  could  be  automated  to  allow  professionals  to  concentrate  on  more 
difficult  and  important  tasks. 

The  lack  of  standardized  tools  has  resulted  in  problems  with  acquisition  and  software  development. 
A  large  number  of  tools  tend  to  be  inaccessible,  difficult  to  use  or  inefficient.  Software  tool  acqui¬ 
sition  could  become  more  definable  and  therefore  more  achievable  with  a  standard  set  of  tools  that 
provide  consistent  computer  support,  particularly  for  software  development  For  example,  high 
order  languages  should  strive  for  machine  independence  that  would  improve  transportability. 

Software  reusability  is  considered  by  many  to  be  a  powerful  means  of  reducing  software  reinvention. 
However,  attempts  to  reuse  software  inappropriate  to  the  given  application  may  actually  hinder 
the  development  process  and  degrade  the  resultant  product. 

Capital  investment  is  needed  to  improve  the  support  environments  and  reduce  problems  such  as 
reinvention  or  the  lack  of  adequate  methodologies  and  tools.  There  are  three  main  problems  asso¬ 
ciated  with  the  lack  of  capital  for  conventional  software  expenditures.  First,  software  is  developed 
on  out-of-date  hardware  not  designed  to  support  the  software  effort.  Second,  schedule  slips  due  to 
the  lack  of  adequate  tools  introduce  higher  overall  costs.  Third,  not  enough  capital  is  invested  in 
IR  A'  D  projects. 

3. 1.2.3  Software  Product 

The  Sofi ware  Product  category  is  defined  as  the  operational  computer  software  and  materials 
nec  essary  to  provide  life  cycle  support.  These  include  requirement  and  design  specifications,  source 
code,  test,  dat  a,  system  generation  data,  unique  support  tools,  etc.  Problems  have  been  experienced 
in  the  following  areas: 
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Doesn’t  Meet  the  Need; 


•  Software  Metrics; 
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•  Design  Attributes; 
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•  Documentation;  and 

1 

•  Immutable  Software. 

‘Doesn’t  meet  the  need’  refers  to  the  users  dissatisfaction  with  the  system.  An  inferior  product  can 
easily  be  developed  as  a  result  of  unclear,  vague  or  incorrect  requirements.  System  performance  is 
highly  dependent  on  the  type  and  quality  of  testing.  In  both  situations,  users  end  up  with  a  system 
which  simply  does  not  work  to  perform  the  mission. 

Software  metrics  provide  essential  analytic  models  and  empirical  data  on  software  to  help  in  the 
selection  of  software  engineering  techniques  for  estimating  development  resources  and  to  evaluating 
future  costs.  However,  few  effective  models  were  found  in  use  to  validate  conventional  life  cycle- 
costs  and  productivity  during  the  development  and  support  phase.  Few  good  analytic  models  and 
methods  are  available  to  gather  empirical  data  to  estimate  future  costs  and  system  impacts  No 
method  to  determine  how  to  develop  firmware  has  been  established.  Lastly,  managers  do  not  receive 
the  status  reports  necessary  for  cost  and  schedule  analysis. 

Design  attributes  deal  with  the  provision  of  an  acceptable  program  solution  to  problems  defined  in 
the  requirements  specifications.  Some  of  the  major  areas  where  design  problems  have  been  identified 
are:  not  adhering  to  good  software  development,  techniques;  inadequately  designed  requirements; 
and  understanding  software  versus  hardware  implications.  Improperly  designed  software  is  often 
the  result  of  not  following  a  defined  software  methodology,  a  lack  of  adherence  to  a  top-down 
hierarchical  system  breakdown  and/or  a  lack  of  consideration  to  human  engineering  as  an  important 
design  issue.  Another  major  design  problem  occurs  when  requirements  are  vague  or  incorrect  and 
the  engineer  makes  the  wrong  assumptions  to  implement  the  requirement.  Also,  correct  software 
design  decisions  depend  on  a  basic  understanding  of  the  hardware  involved.  Since  conventional 
software  normally  cannot  handle  system  modifications  without  added  cost  expenditures  or  schedule 
delays,  ineffective  system  design  severely  hinders  the  system’s  success. 

A  major  problem  affecting  conventional  software  documentat  ion  is  that  financial  resources  allocated 
to  its  preparation  do  not  necessarily  reflect  actual  costs.  Therefore,  whenever  a  schedule  slips,  or 
the  budget  is  overspent,  the  documentation  effort  is  relaxed.  Consequently,  successive  software 
changes  are  not  reflected  in  the  documentation  Another  problem  stemming  from  a  reduction 
in  documentation  is  the  determination  of  what  needs  to  be  documented.  For  instance,  system 
interfaces  must  be  documented  and  the  documentation  of  requirements  and  design  is  important 
Also,  traceability  between  documents  is  critical  but  generally  not  possible  because  of  unavailable 
tools. 

One  of  the  problems  with  the  development  of  conventional  computer  software  is  that  it  is  generally 
tailored  to  the  specific  application  or  hardware  environment.  Consequently,  software  packages  are 
often  system  unique,  non-portable  and  non-reusable  (Immutable  Software)  Acquisition  agencies 
arc  then  forced  to  repeatedly  pay  for  software  which  could  have  been  available  elsew  here. 
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3.2  AI  Software  Approach 


3. 1.2.4  People 

Qualified  software  personnel  for  managerial  and  technical  positions  are  required  u-  avoid  project 
problems.  People  problems  exist  in  the  following  subcategories: 

•  Skills; 

•  Availability;  and 

•  Incentive. 

The  rapid  spread  of  digital  technology  has  resulted  in  a  widespread  shortage  of  skilled  system  en¬ 
gineers,  software  engineers  and  managers.  Consequently,  the  three  main  problems  of  professional 
skills,  professional  availability  and  professional  incentive  have  been  identified  A  highly  valued  and 
respected  managerial  skill  is  the  ability  to  guide  software  through  the  life  cycle  from  requirements 
to  maintenance  However,  the  skills  necessary  to  do  this  requires  a  broad  range  of  software  experi¬ 
ence  and  acquired  knowledge.  Few  educational  programs  exist  to  provide  piof-.-ssionals  with  these 
necessary  skills  Therefore,  the  lack  of  available,  experienced  and  skilled  software  professionals 
continues  to  exist.  Professional  incentives  are  needed  to  attract  and  retain  expert  software  profes¬ 
sionals  A  lack  of  reward  for  excellence  and  the  competitive  software  market  in  search  of  skilled 
software  professionals  results  in  high  turnover  rates. 

3.2  AI  Software  Approach 

A1  techniques  have  typically  been  oriented  towards  ill-structured  problems  with  solutions  that  are 
characterized  as  heuristic,  non-algorithmic  and  non-detorministic.  Al  technology  application  areas 
I  hat  reflect  these  characteristics  include  robotics,  vision,  natural  language,  automatic  programming, 
planning  and  expert  systems.  The  Al  solution  approach  to  these  areas  is  in  contrast  to  conventional 
software  methods  that  generally  deal  with  well-defined  algorithmic  problems  and  deterministic 
solutions. 

The  characteristics  associated  with  AI  applications  have  had  a  strong  influence  on  the  approaches 
taken  to  Al  software  development.  Uncertainty  of  tasks,  knowledge,  results  and  functionality  have 
prompted  A I  programs  to  be  written  to  gain  a  better  understanding  of  the  problem  domain.  Because 
of  this  uncertainty,  rapid  changes  in  the  solution  approach  are  expected  as  more  is  learned  about  the 
problem  domain.  This  has  led  to  the  development  of  powerful  tools  and  integrated  environments  for 
the  development  of  AI  software.  The  result  is  that  AI  development  approaches  are  iterative  in  nature 
and  accomplished  by  small  development  teams  using  highly  integrated  development  environments 

The  Al  technology  environment  that  has  supported  the  non-algorithmic,  heuristic  approach  to 
problem  solving  includes  programming  languages,  development  methodologies,  and  development 
environments  These  technology  areas  have  been  prime  contributors  in  supporting  small  develop¬ 
ment  teams  in  their  quest  to  handle  large  problem  domains 

A I  programming  languages  inherently  support  development  under  uncert  ainty.  The  longest  sur¬ 
viving  Al  language  is  L1SIV  It  supports  delayed  committment,  and  language  extensions  as  two 
approaches  to  managing  uncertainty. 


3.3  Generic  Description  of  tin-  A I  Development  I  t  ocess 


Development  methodologies  are  closely  tied  to  the  iterative  problem  solving  approach  to  \1  soft¬ 
ware.  Both  the  iteration  cycle  and  the  need  to  delay  decisions  or  specifications  until  development  of 
a  mature  prototype  is  complete  are  inconsistent  with  conventional  software  development  method¬ 
ologies.  AI  software  development  methodologies  typically  begin  in  the  middle  of  a  problem  domain 
and  spiral  out  to  a  complete,  acceptable  solution  rather  than  adopting  a  top-down  hierarchical 
decomposition  approach  typical  of  conventional  software  methodologies. 

AI  software  development  environments  reflect  a  committment  to  personal  computing  These  en¬ 
vironments  absorb  some  of  the  tedious  and  routine  work  such  as:  garbage  collection;  memory 
allocation;  integrated  editing,  debugging  and  inspection;  and  documentation  They  also  allow  han¬ 
dling  of  knowledge  based  facilities.  Background  processes  in  AI  development  environments  provide 
tracking  information  about  programs,  display  presentation  of  the  information,  and  provide  active 
agents  to  recognize  conditions,  propose  action  and  perform  cleanup.  MSI*  machines  and  A I  shells 
are  examples  of  such  development  environments. 

3.3  Generic  Description  of  the  AI  Development  Process 

AI  system  development  has  been  described  as  a  highly  iterative  process  and  generally  resembles 
a  build  a  little,  lest  a  little  approach  with  numerous  feedback  loops.  Close  examination  of  the  A I 
development  process  reveals  a  set  of  activities  that  are  integral  to  the  development  of  an  A I  system. 
These  activities,  which  are  a  synopsis  of  the  models  to  follow,  include 

•  Problem  Definition, 

•  System  Design, 

•  Implementation  and  Testing;  and 

•  Support. 

Coupling  these  activities  with  the  concept  of  building  system  increments  allows  lor  the  delivery 
of  partial  capabilities  for  use  early  in  the  development  process  Experience  through  use  can  then 
be  fed  back  into  the  development  process  so  updates  can  be  made  to  future  increments  prior  to 
delivery  for  held  use 

Several  of  (he  better  documented  methodologies  are  described  m  the  following  paragraphs  !  Iiese 
methodologies  recognize  the  iterative,  incremental  approach  to  AI  system  development  The  generic 
descriptions  diMer  somewhat  in  definition  and  phasing,  however,  they  are  all  generally  supportive 
of  the  previously  identified  activities  that  are  applied  in  an  incremental  maniu  r 

3.3.1  The  Expert  System  Model 

In  Hutlilunj  Erpcrt  Systems  [35,  Hayes  Rothj  the  authors  delme  the  major  stages  of  knovvlcdgi 
acquisition  for  expert,  system  construction  as 

•  Identification; 


f  s' 


3.3.2  The  DEC  Model 


Concept  ualizat  ion 
Formal  i/at  ion ; 
Implement  at  ion  ,  and 
Testing 


V-. 


rcl 


Although  defined  in  terms  of  knowledge  acquisition,  the  stages  a  iso  represent  how  an  expert  system 
is  constructed  Each  stage  defines  a  set  of  activities  related  to  expert  system  development  As 
the  authors  point  out,  these  stages  are  not  sequential  in  nature  but  instead  overlap  each  other 
Further,  development  of  an  expert  system  does  not  constitute  a  single  pass  through  the  stages 
There  is  continuous  feedback  from  each  of  the  stages  to  all  prececding  stages  thus  representing  the 
incremental  nature  of  the  KBS  knowledge  acquisition  process. 

Further  examination  of  the  stages  of  knowledge  acquisition  reveals  a  manning  between  stages  and 
the  previously  defined  phases  of  Problem  Definition,  System  Design,  Implementation  and  Testing, 
and  Support  Identification  equals  Problem  Definition  and  involves  defining  the  problem  domain, 
developing  informal  descriptions  of  the  problem  and  partitioning  it  into  su 'problems  Conceptu¬ 
alization  and  Formalization  parallel  the  System  Design  phase  activities  of  knowledge  acquisition 
for  a  selected  subset  of  the  problem  domain  and  developing  a  partial  specification  detailing  the 
tools  and  representations  thought  to  he  appropriate  for  the  problem  domain  Implementation  and 
I’esting  includes  the  actual  construction  of  the  KBS  system  or  its  subset  in  the  chosen  language 
environment  followed  by  a  test  of  its  correctness. 

The  five  stages  are  further  mapped  into  two  phases  rather  than  the  four  activities  identified  above. 
'File  two  phases  are 

•  F’hase  I  -  Identification  and  Conceptualization 

•  Phase  2  -  Formalization,  Implementation  and  Testing 

The  differences  in  phasing  do  not  represent  a  conflict  but  instead  offer  two  different  views. 

3.3.2  Tho  DEC  Modol 

In  rhr  Artifirnl  lnl>  lliqcm  r  Erpi  ru  nrr:  An  Introduction  |59,  Scown!  .  the  author  defines  a  model 
for  expert  system  development  which  essentially  includes  four  phases  These  phases  are: 

•  Problem  Identification: 

•  Functional  and  Design  Specification, 

•  ('rente  Bounded  I’roiotvpo,  and 

•  Incremental  l)e\e|opmrnl 

3  8 


A 


vV 

->>> 

-V-V- 


.\v. 


I 


3.3.3  The  Di})in<‘tcr  Advisor 


Problem  Identification  includes  the  definition  of  the  problem  domain  I'uiict  ional  and  Design  Speci¬ 
fication  equates  to  the  previously  generic  phase  of  System  Design  wherein  a  subset  of  the  knowledge 
domain  is  defined  and  a  functional  specification  for  a  bounded  prototype  is  developed  In  addition, 
a  design  specification  is  developed  detailing  the  tools  and  representations  thought  to  lie  appropriate 
to  the  problem  domain.  Creation  of  the  bounded  prototype  represents  Implementation  and  Testing 
for  the  first  deliverable  product.  Incremental  Development  represents  all  subsequent  development 
and  testing  activites  associated  with  deliveries  on  an  incremental  basis  in  the  evolutionary,  iterative 
environment  of  AI  system  development 

The  DEC  model  closely  t  racks  the  generic  model  previously  defined.  The  development  model  v  iews 
the  process  as  a  set  of  sequential  activities  to  the  point  of  achieving  a  bounded  prototype  At  this 
point  it  becomes  the  iterative  process  of  develop  an  increment,  test  it  and  then  release  it  to  the 
field  for  further  feedback  and  support. 


3.3.3  The  Dipmeter  Advisor 

The  Dipmeter  Advisor  expert  system  experiences  reflect,  two  different  aspects  of  phasing  consider¬ 
ations  [62,  Smithj.  One  reflects  an  oscillation  that  occurred  during  the  project  and  the  other  the 
phased  approach  taken  to  develop  successive  prototypes  for  the  Dipmeter  Advisor 

The  oscillation  is  best  explained  by  examining  the  two  phased  development  process  that  occurred 
in  respect  to  tool  development  and  prototype  implementation  The  first  phase  constituted  a  fea¬ 
sibility  demonstration  in  which  knowledge  was  collected  for  a  constrained  problem  and  the  tools 
selected  appropriate  to  the  domain.  The  second  phase  included  an  expanded  implementation  of 
the  domain  knowledge  where  the  expert  system  development  tools  remained  relatively  constant 
As  development  progressed,  points  in  time  were  reached  where  the  initially  defined  tools  did  not 
allow  further  expansion  of  system  expertise.  The  development,  process  was  iut errupt  ed  at  t  his  point 
while  the  phase  one  activity  was  restarted  to  construct  a  more  powerful  set  of  tools 

The  Dipmeter  Advisor  project  went  through  a  three  phase  development  proms-  ol. 


feasibility  Demonstration 


Utility  Performance  Demonstration 


I  Itility  /  Performance  Evaluation 


The  f'ea  ability  phase  included  the  activities  of  knowledge  acquisition  and  prototype  implementa¬ 
tion,  and  as  such  really  could  be  conceived  of  as  two  separate  phases  1  he  I  tility  Performance 
Demonstration  phase  examined  the  product  for  its  ability  to  provide  the  correct  answers  but  also 
examined  tin  viability  of  the  prototype  as  a  commercial  product  1  In  Utility  I Yrform,iiice  l.\al- 
untiou  phase  consisted  of  field  evaluation  of  the  prototype  All  of  these  phase-  included  pr  Mem 
and  enhancement  feedback  (or  preparation  of  the  next  prototype 
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3.3.4  Other  donor ic  Models 


3.3.4  Other  C-ipiirrir  Models 


There  arc  numerous  oi  lier  models  which  have  been  developed  which  in  some  way  parallel  one  of  the 
previous  mentioned  models  j‘Jd.'2'2,  Harmon,  (latesj  These  models  relic.  I  i>o.l.-.iii,g  which  is  Iiol h 
overlapping  and  iterative  in  nature,  and  an  approach  that  relies  on  extensive  prototyping. 


(ionoric  Definition  of  KBS  Developments 


Defining  a  j  enenr  description  of  the  KBS  development  process  is  a  task  i.hai  requires  the  identifi¬ 
cation  of  tin  steps  or  phases  in  the  process  and  associating  activities  \vtil<  each  phase  The  iterative 
nature  of  K  US  developments  further  complicates  any  visualization  of  t  he  pro  ess  and  requires  that 
one  think  about  it  as  a  repeating  process  The  generic  phases  of  Problem  Identification,  System 
Design.  Iiiq  lementation  and  Testing,  and  Support,  as  summarized  in  Tabic  .1 . 3  ">  1,  represent  a 
phased  putt  rn  (hat  has  been  largely  substantiated  by  the  available  literal. .re  and  one  that  was 
used  as  a  x|  irMng  point  for  the  modeling  effort. 


Table  3..'!. 5  I:  Generic  Mode!  Phases 


Mode! 


Problem 
Definii  i<  pii 
System  Design 


Implement  at  ion 
and  Test  mg 


Expert 

System 

Model 

Identification 

Conceptualization 


D  EC- 
Model 

Problem 
Identification 
Functional  u;id 


Formalization  Design  Specification 


Dipmeter 

Advisor 

Model 


Knowledge 
Acq  lisii  in; 


Marinon 
and  King 
Model 

Problem 
Selection 
Prototype 
Development  and 
System  Design 


Implementation 

Testing 


Bounded 
Prototype  and 
Increment  al 
Development 


Delivery 


Prototype  Delivery  of 

Implementation  anil  Computer  System 
Demonstration  of  and  System 

Utility  and  Evaluation 

Performance 


Evalual  ion 
of  Utility  and 
Performance 


Integration 
and  Maintenance 


The  activities  associated  v>ilh  each  phase  have  been  widely  documented  m  the  literature  and  ac¬ 
cepted  by  orj,  im/,il  e  ns  w  it  h  sigmlii  ant  expert  systems  development  activities  Tim  <  Imseii  generic 
description  provubs  ’lu  foundation  for  the  discussions  and  data  represented  in  the  remainder  of 
I  he  rej  :  e  i 
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3.4  Description  of  KBS  Development  Characteristics  iSfiC 

There  are  many  characteristics  associated  with  a  KBS  developmmi  project  M* of  i  char¬ 
acteristics  either  have  no  counterpart  or  are  exercised  diU'erentlv  w  hen  compared  (o  conventional 
software  development  activities.  In  this  subsection,  detailed  descriptions  are  provided  for  the  fol¬ 
lowing  KBS  features: 


1.  Knowledge  acquisition; 

2.  Knowledge  representation; 

3.  Reasoning  methods; 

4.  Redundancy  exploitation; 

5.  Development  environment; 

t>.  Exploratory  programming  style; 

7  Rapid  prototyping: 

8.  Small  development  teams; 

9.  User  involvement; 

10.  Documentation  produced; 

1 1.  Testing;  and 

12.  Management  control  mechanisms 

The  manner  in  which  the  above  features  relate  to  conventional  software  development  i  haracterist  i< 
should  be  evident  from  the  discussion  that  follows. 
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3.4.1  Knowledge*  Acquisition  • 

,v>;: 

A  major  aspect  of  the  development  of  expert  systems  resides  in  knowledge  acquisition  The  process 

involves  the  transfer  and  transformation  of  problem-solving  expertise  from  a  knowledge  source  !-*v'v 

such  as  human  experts,  data  bases,  and  literature,  to  a  program  There  is  no  such  activity  in 

the  algorithmic  world  of  conventional  software  development  because  knowledge  is  not  explicitly 

represented  as  a  separate  system  component  Unfortunately,  know  ledge  acqmst  uut  represents  the 

main  biittleneck  in  expert  system  development  because  of  the  difficulty  with  exit  e  ling  knowledge 

for  incorporation  into  a  knowledge  base  As  a  result,  system  development  linn  o  prolonged 

Another  problem  is  based  on  the  fact  that  only  a  limited  number  of  automated  '  U  ■  xot  to  aid 

in  the  at  qmsil  ion  of  know  ledge  from  t  he  domain  expert  I  hen-lore,  kin  wh-h;.  i  oie-ii  ion  m  pen  is 

mi  the  skill  of  the  knowledge  engineer  who  must  effectively  <  omui'inn  al  ■  witli  the  d  main  Xpert  . — 

in  order  to  understand  and  structure  the  knowledge  Knowledge  acqmoiion  miglii  involve  -"veral 
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3.4.2  Knowledge  Representation 


planned  interviews  between  the  knowledge  engineer(s)  and  the  donw.m  experts;  •>i!t  >  purpose 

of  extracting,  summarizing  and  struct uring  the  knowledge 

Knowledge  acquisition  begins  with  an  identification  of  the  exp<  rt  system's  part  vqiants.  goals  and 
problems  <  tone  rally,  one  knowledge  engineer  is  assigned  to  research  the  system  s  ,•  now  ledge  domain 
and  hold  interviews  Likewise,  one  domain  expert  usually  represents  tin  expert  system  needs 
However,  depending  on  the  scale  and  complexity  of  the  solution  space,  tn<,r-'  than  one  engineer 
or  expert  mav  be  assigned  to  a  project.  The  knowledge  engineer’s  respnnsdn.iMcs  to  research  the 
domain  and  schedule  meetings  with  the  expert  is  designed  to  aid  n;  the  proi  c  f  identifying  the 
system's  problems  and  goals 

An  inherent  difficulty  with  knowledge  acquisition  lies  in  helping  the  expert  to  structure  his/her 
knowledge,  to  identify  and  to  formalize  domain  concepts.  Domain  experts  cannot  always  logically 
express  the  manner  in  which  they  solve  problems.  Therefore,  a  number  of  interviewing  techniques 
are  available  to  the  engineer  to  help  measure  performance  and  uncover  expertise  Audio  tapes  of 
the  interviews  are  not  uncommon 

As  key  problem-solving  methods  are  identified,  a  means  to  produce  and  mime  he  expert’s  knowl¬ 
edge  generally  follows  Two  well-known  methods  are  to  either  transi  ribe  Cue  problem-solving 
knowledge  onto  paper  or  to  produce  a  prototype  The  motivation  behind  ,»oth  methods  revolves 
around  a  way  to  present  the  expert  with  a  visual  aid  which  allows  him/her  to  mentally  step  through 
the  process  to  verify  the  validity  of  the  problem-solving  technique  Essentially,  gathering  an  ex¬ 
pert  ’s  knowledge  involves  an  incremental  approach  to  define  and  redefine  problems  until  the  domain 
know  ledge  contains  an  adequate  amount  of  information  to  respond  appropriately  to  a  problem. 

The  main  idea  which  drives  the  knowledge  acquisition  process  from  iden1  ifirat ion  to  conceptualiza¬ 
tion  of  knowledge,  revolves  around  the  need  to  formalize  the  concepts.  piobi-ms  and  information 
flow  into  a  more  formal  knowledge  representation  which  defines  the  data  structures,  inference  rules 
and  control  strategies  A  recognized  dilemma  with  knowledge  acquisition  resides  in  identifying 
an  expert  whose  knowledge  adequately  reflects  the  problem  domain  Apparently,  an  incremental 
approach  to  software  acquisition,  in  the  form  of  several  prototype  releases,  resolves  the  conflict 
of  whether  data  gathered  is  appropriate  to  the  system’s  needs  The  approach  allows  the  domain 
expert  an  opportunity  to  watch  the  system  work  which  enables  him/her  to  comment  on  erroneous 
system  problem-solving.  In  turn,  the  knowledge  engineer  is  provided  with  a  tool  with  which  to 
identify  problems  areas  such  as:  what  kind  of  knowledge  is  not  sufficiently  defined;  what  kind  of 
logic  needs  to  be  redefined;  and  what  new  areas  of  knowledge  should  be  acquired. 

Although  the  availability  of  tools  to  aid  in  the  knowledge  acquisition  process  is  very  limited,  some 
tools  do  exist  (See  Section  3  4  5  for  a  list  and  explanation  of  these  tools)  For  example,  there  are 
tools  that  help  construct  and  refine  expert  systems.  Although  the  tools  do  not  acquire  rules,  they 
sometimes  allow  the  expert  to  define  and  organize  the  rule  building  blocks. 

3.4.2  Knowledge  Representation 

Know  lediv  ai  quoit  ioii  and  knowledge  representation  are  closely  intertwined,  nouseqiienl  ial  pro¬ 
cesses  Throughout  ill"  development  period  of  a  knowledge  based  system,  these  steps  are  revisited 
many  times  to  initially  define  and  then  refine  the  knowledge  base.  In  designing  a  knowledge  based 
system,  it  is  not  an  easy  'ask  to  .determine  how  the  knowledge  can  best,  be  represented  to  guide 
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the  system’s  problem  solving  behavior.  Nonetlieless,  m  order  lo  extract  knowledge  from  an  expert, 
it  may  be  helpful  to  understand  apriori  how  the  knowledge  will  be  represented  However,  with¬ 
out  already  having  the  knowledge,  it’s  not  even  clear  what  has  to  be  represented  ( 'onsc(|uent  ly , 
ill  typical  systems,  a  relatively  small  but  central  fraction  of  the  domain  expertise  is  collected  by 
the  knowledge  engineer  interacting  with  the  expert  over  a  short  period  of  time  (i.e.  one  to  two 
months).  During  this  period,  the  knowledge  engineer  has  a  chance  to  evaluate  the  knowledge, 
observing  patterns  and  levels  of  abstractions  before  implementation  decisions  are  made, 

How  knowledge  is  structured  in  a  program  is  highly  dependent  on  the  conceptual  framework 
whether  or  not  knowledge  is  centered  around  objects  or  processes;  or  thought  of  symbolically. 
Since  a  well-defined  knowledge  representation  methodology  can  simplify  complex  problems,  under¬ 
standing  knowledge  in  order  to  choose  from  a  large  variety  of  available  techniques  becomes  vital 
The  following  subsections  examine  four  commonly  referenced  techniques  to  represent  knowledge: 
semantic  networks,  frame-based,  rule-based  and  logic  programming  methods 

3.4.2. 1  Semantic  Net 

Semantic  net,  a  knowledge  representation  method  based  on  a  network  structure,  consists  of  points 
called  nodes  that  are  connected  by  links  referred  to  as  arcs.  The  links  define  the  relationship  bet  ween 
the  different  nodes.  Each  node  in  the  semantic  net,  generally  represents  physical  or  conceptual 
concepts,  events  or  objects.  A  variety  of  choices  exist  for  the  definition  of  arcs,  depending  on 
the  knowledge  represented.  Some  of  the  more  familiar  arcs  used  to  represent  the  hierarchical 
relationship  between  nodes  are  i*-a  and  ha*- part.  Natural  languages  tend  to  use  arcs  such  as  <itp  ut. 
object  And  recipient.  As  a  simple  example  from  the  statements:  'human  is  a  mammal' and  ‘man  is  a 
human’,  one  can  infer  that  man  is  a  kind  of  mammal  if  the  semantic  net  represents  the  know  ledge 
as  a  hierarchical  breakdown  where  man  is  a  subset  of  human  and  human  is  a  subset  nT  mammal 

Some  common  mechanisms  or  features  of  semantic  nets  are  inheritance  and  demons.  Inheritance  is 
the  ability  of  one  node  to  adopt  propert  ies  from  nodes  at  a  higher  level  on  a  hierarchy  A  properl  y 
inheritance,  which  can  be  implied  from  an  is.a  relationship,  means  that  instances  oT  a  class  can 
have  all  characteristics  of  classes  to  which  they  belong.  Thus,  man  is  an  instance  of  human  and 
inherits  properties  of  human  which  is  a  kind  of  mammal.  Naturally,  redundant  information,  ana 
therefore  wasted  storage  is  avoided.  Demons  are  another  aspect  of  semant  ic  nets  The  main  Incus 
of  demons  is  to  provide  a  resource  of  information  which  can  be  used  as  values  w  henever  net  led  or 
specifically  requested  of  t  he  database. 

3. 4. 2. 2  Frames 

Frames  are  another  method  of  represent  mg  knowledge  (facts  and  relat  lonships)  Similar  to  sonant  ic 
nets,  frame-based  knowledge  representations  make  use  ol  a  network  of  nodes  connected  by  relations 
into  a  hierarchy .  Frames  differ  from  semantic  nets  in  that  nodes  are  defined  by  groups  of  at  1  •  dun es 
(also  referred  to  as  slots)  In  turn,  each  slot  may  contain  attribute  values  default  values,  p  inters 
toother  frames,  sets  of  rules,  procedural  attachments  (that  execute  whenever  attribute  values  are 
referenced),  etc.  Basically,  top-level  nodes  in  the  hierarchy  represent  general  concept-  and  lower 
nodes  refer  to  specific  inst  ances  of  concepts  and  therefore  inherit  properties..!  higher-level  nodes 
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3.4.2  Knowledge  Representation 


The  procedural  attachment  allows  for  two  complementary  ways  to  ‘•lute  ami  More  'um  a  pi  - 
cedtiral  or  declarative  representations.  Declarative  representations  are  exp, test  I,  -se  they  are 
simply  an  assertion  about  a  fact  In  contrast,  procedural  representations  are  more  difficult  to 
define  because  facts  are  specified  by  how  they  are  used. 

The  power  of  frames  consists  in  an  ability  to  efficiently  combine  doclaratm  will;  procedural  rep¬ 
resentations.  Because  of  their  modularity,  declarative  representations  art  more  easily  maintained 
and  adaptable  to  independent  and  changing  facts.  Procedural  attachments  are  m  >re  efficient,  hut 
more  difficult  to  maintain 


3. 4. 2. 3  Rules 


Rules  are  another  form  of  knowledge  representation,  based  on  IF  condition  TURN  ..  bvni  state¬ 
ments.  Basically,  when  t  he  facts  stipulated  in  the  IF  part  of  the  rule  are  true  the  TURN  action  part 
of  the  rule  is  executed  When  this  happens,  the  rule  is  considered  (i’c J  or  om.-.f  d  Results  may 
he  manifested  in  an  instruction  for  the  system  to  add  a  new  hypothesis  to  \\e  database,  program 
control,  or  peripheral  device  control. 

The  pairing  of  t  he  II  portions  of  rules  to  facts  result  in  mf<  muy  <•/ i,in>«  T!  e  inference  chain  shows 
how  the  system  uses  rules  to  reach  its  conclusions. 

There  ire  basically  two  inference  mechanisms  (ways  in  whirl)  rules  can  he  set  up  in  a  rule-based 
system):  forward  chaining  and  backward  chaining.  Forward  chaining  refers  to  the  gathering  of 
pieces  of  information  in  an  attempt  to  build  forward  to  an  end  goal.  Backward  chaining  begins 
with  a  goal  and  works  backward  to  seek  a  chain  of  premises  that  actounts  for  all  the  facts  at  hand. 

Rules,  as  a  knowledge  representation  scheme,  provide  a  natural  environment  for  depicting  processes 
driven  by  rapidly  changing,  complex  environments.  Rules  not  only  provide  spr cifications  on  how  a 
program  should  react  in  relation  to  changing  data  without  necessarily  knowing  the  flow  of  control, 
but  can  also  easily  explain  what  a  program  did  or  how  a  conclusion  was  reached. 


3. 4. 2. 4  Logic  Programming 

Logic,  a  fourth  ty  pe  of  knowledge  representation,  encompasses  a  variety  of  logical  notations  and  is 
based  on  the  model  of  proof  in  mathematical  logic.  Two  common  notations  are  propositional  logic 
and  predicate  calculus. 

Propositional  logic  deals  with  statements  that  are  either  true  or  false.  Each  statement  ran  be 
linked  together  bv  n  connective  such  as  and,  or,  not,  unpins  and  equivah  nt  which  result  in  a 
compound  statement  Rules  exist  to  propogate  the  truthfulness  of  compounds  Other  rules  support, 
inferem  ing 

Predicate  <  alculns  is  considered  an  extension  to  propositional  logic.  Elementary  units  are  referred 
to  a«  objects  and  statements  about  objects  are  considered  predicates.  Therefore,  the  statement, 
is  son( Brin  e, Tom)  is  a  two- plate  predicate  with  two  objects,  Bruce  and  Tom.  The  predicate  can 
be  evaluated  as  a  true  nr  false  assertion  that  'Ibm  is  or  is  not  the  son  of  Bruce  Predicates  can  be 
linked  into  larger  expressions  by  means  of  the  same  connectives  used  in  propositional  logic. 


» r-\r.  it,  vK  v;v;  v,  v.  ^  7-."^."-".-v-.^.~.r.-^.  -^.-.-.-^,-y-y.-»^ 


3 .4.3  Itr.ismiing  Mr t hods 


PROLOG,  which  stands  fur  PROgramming  language  fi  r  l/H.e  .  is  a  <  lassie  example  of  a  language 
which  incorporates  some  of  the  principles  of  predicate  logic  I  lie  PR()L()(i  conirol  slruciure  is 
logical  inference.  Basically,  one  states  facts  and  rules  about  objects  and  PROLOti  can  deternnne 
whether  a  specific  conclusion  can  be  deduced  given  the  collected  data 


3.4.3  Reasoning  Methods 

There  are  numerous  reasoning  methods  reported  in  the  literature  l ‘onsequent  ly.  the  discussion  in 
this  section  is  generally  limited  to  the  more  common  strategies  and  those  methods  reported  on  in 
the  case  studies  presented  in  Section  4 

Three  areas  of  reasoning  that  might  guide  a  KBS  system  through  its  knowledge  base  include 


•  inference  strategics; 

•  inexact  reasoning;  and 

•  control. 

Inference  strategies  are  generally  based  on  the  use  of  logical  axioms  The  most  common  inference 
strategy  used  in  knowledge  systems  today  is  the  application  of  a  logical  rule  called  inoihi<  / mm  n< 
(29,  Harmon)  The  method  states  that  if  the  premise(s)  of  a  rule  is  true,  then  the  eonelusioii(s)  is 
true.  Namely,  when  A  is  known  to  be  true  and  an  axiom  ‘‘If  A,  then  B'  exists,  it  is  valid  to  conclude 
that  B  is  true.  The  application  of  modus  potions  has  two  implications  first,  the  method  is  simple 
so  that  reasoning  based  upon  it  is  easily  understood  Secondly,  not  all  valid  conclusions  can  be 
drawn  with  the  use  of  modus  ponens  alone.  Modus  It >1  h  as  can  be  used  to  enrich  the  infen  neing 
strategy.  This  logical  axiom  slates  that  if  B  is  known  to  be  false  and  an  axiom  If  \.  then  B 
exists,  then  it  is  valid  to  conclude  that  A  is  false. 

Resolution  is  another,  more  general  inference  rule  The  basis  of  resolution  is  that  if  there  are  two 
axioms  of  the  form: 
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•  A  or  B 

•  not,  B  or  C 

then  A  or  C  logically  follows  The  expression  A  or  (’  is  called  the  resolvent  ,  f  i  he  lirsi  two 
expressions.  Resolution  can  be  generalized  so  that  there  can  be  any  number  of  disjoin  ts  in  either 
of  the  two  initial  expressions  as  long  as  one  expression  has  a  disjunct  that  is  the  negation  of  a 
disjunct  in  the  other  expression  In  the  general  case,  conclusions  drawn  in  logical  axioms  ^  u  h  as 
modus  ponens  and  modus  lollens  can  also  be  arrived  at  using  resolution 

Because  a  knowledge  base  may  not  necessarily  contain  all  the  information  required  to  I'.eh  a 
conclusion,  inferencing  techniques  must  include  reasoning  about  uncertainty  In  oilier  wor.  s.  like 
an  expert,  tin1  inference  engine  must  be  able  to  deal  with  inronipb  t  e  informal  mu  ini  x .n  t  re. a  .oiling 
has  been  implemented  using  either  numerical  or  non-numeric, ll  techniques 
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Numerical  methods  may  include  the  use  of  confidence  levels  or  certainty  fad  or?  a- inched  to  a 
particular  conclusion  and  propogated  as  further  inferences  are  made.  For  instance,  if  u  man  is 
wearing  a  white  collar,  he  must  be  a  priest  may  be  associated  with  a  30 /c  confidence  level. 

Another  numerical  approach  to  dealing  with  uncertainty  is  the  use  of  Bayesian  decision  theory. 
Simply  stated,  this  method  associates  a  subjective  probability  value  with  every  hypothesis  to  mea¬ 
sure  the  degree  of  belief.  In  the  absence  of  evidence,  any  hypothesis  is  assumed  to  have  an  initial 
probability  which  changes  as  evidence  is  gathered.  The  ultimate  goal  is  to  choose  the  hypothesis 
with  the  highest  probability. 

The  fundamental  concern  with  numerical  methods  of  handling  uncertainty  that  they  hide  the 
reasoning  that  produces  them.  [14,  Cohen],  In  addition,  while  numbers  are  <-asy  to  propogate 
over  inferences,  what  the  numbers  mean  may  not  be  clear.  The  theory  of  endorsements  is  a  non- 
numerical  method  of  inexact  reasoning.  Because  inexact  reasoning  is  a  knowledge  intensive  task, 
the  theory  of  endorsements  employs  a  heuristic  approach  to  dealing  v  ith  uncertainty,  endorsements 
are  records  of  information  that  afTect  one’s  certainty,  including  the  kind  of  evidence  available  and 
the  methods  used  to  produce  the  current  hypothesis  from  uncertain  preconditions  Endorsements 
are  propogated  over  inferences  using  heuristics  in  a  manner  that  is  sensitive  to  the  context  of  the 
inference.  Furthermore,  endorsements  can  be  ranked  in  an  effort  to  choose  the  hypothesis  with  a 
superior  level  of  endorsement. 

Endorsement  is  similiar  to  recording  justifications  in  a  truth  maintenance  system  (TMS)  [14,  Cohen] 
The  crucial  difference  is  that  in  a  TMS,  a  justification  is  used  to  decide  whether  a  conclusion  has 
support.,  hut  the  kind  of  support  (or  evidence)  is  irrelevant 

Both  endorsements  and  TMS  support  nonmonotonic  reasoning  wherein  con  elusions  that  are  true 
at  one  instance  may  need  to  be  retracted.  Nonmonotonic  systems  generally  include  a  dependency 
network  approach  to  logic  which  maintains  dependencies  between  values  'IVuth  values  are  pro¬ 
pogated  using  constraints  supplied  by  logical  expressions.  With  dependency  networks,  one  can 
keep  track  of  justifications  made  for  all  conclusions  which  can  then  he  modified  if  previous  assump¬ 
tions  are  withdrawn.  For  example,  in  a  planning  system,  it  may  make  sense  to  proceed  in  a  certain 
way.  As  more  information  becomes  available,  an  earlier  derision  may  need  to  be  retracted.  As  a 
consequence,  all  the  implications  based  on  that  initial  decision  also  need  to  be  retracted. 

In  a  monotonic  reasoning  system,  all  values  concluded  remain  true  throughout  the  course  of  the 
program  run.  In  other  words,  farts  that  become  true  remain  true. 

In  addition  to  inference  strategies  and  inexact  reasoning,  their  are  two  primary  problems  associated 
with  a  knowledge  based  system  that  are  addressed  by  the  control  portion  of  the  inference  engine: 

•  where  to  begin  the  reasoning  process;  and 

•  what  to  do  when  alternative  lines  of  reasoning  emerge. 

The  first  problem  can  be  answered  by  the  use  of  forward  and/or  haekwanl  i  limning  mentioned  in 
Section  3  1  1’  3  If  the  possible  outcomes  are  known  and  if  they  are  reasonably  small  in  number, 
then  a  backward  chaining  or  goal  directed  approach  is  very  efficient.  On  the  other  hand,  if  the 
number  of  possible  outcomes  is  large  or  if  the  possible  goal  states  are  not.  known  at  the  outset,  a 
forward  chaining  or  data  driven  approach  is  needed. 


wv 

Rf, 

py 

V 


5sw 

]i 


v-  .>  v.v >. - » > . - . - .  ■,  o  v  v  v v  v "jOl 


3.4.4  Redundancy  Exploitation 

Planning  islands  represent  another  approach  to  the  first  problem  Namely,  begin  processing  where 
there  is  the  most  information  and  the  least  amount  of  uncertainly. 

The  second  problem  can  be  addressed  by  deciding  on  either  a  ihpth  ju  st  or  hr<  mllti-lo  *1  search 
With  the  more  common  method,  depth-first,  the  inference  engine  focuses  on  searching  for  detail 
and  descending  to  deeper  levels  to  produce  a  subgoal.  A  breadth-first  search  sweeps  across  all 
possible  premises  in  a  rule  before  digging  for  greater  detail 

In  addition  to  depth-first  and  breadth-first,  there  are  many  more  search  strategies  that  have  been 
employed  in  KBS  system  developments.  Nonet  heless,  a  discussion  of  additional  strategies  is  beyond 
the  intent  of  the  discussion  herein. 

With  any  of  the  control  strategies  discussed  above,  the  inference  engine  can  provide  the  basis  for 
an  explanation  facility  by  keeping  track  of  where  it  went  to  form  any  particular  coin  lusion 

3.4.4  Redundancy  Exploitation 

One  of  the  features  of  KBS  development  that  has  impact  on  performance,  reliability  and  pialit  . 
is  the  redundancy  of  information  and  processing  This  characteristic  is  in  part  due  to  the  nature 
of  the  more  common  knowledge  representation  schemes  and  inference  mechanisms  and  m  part  to 
the  “middle-out”  problem  solution  approach  Typically,  each  item  is  evaluated  or  tested  -cvcral 
times  before  all  conditions  and  changes  are  satisfied.  KBS  systems  are  generally  built  recursively 
so  a  module  may  be  used  or  executed  for  a  variety  of  purposes.  Use  oT  both  forward  and  bai  kward 
chaining  in  the  same  system  provides  redundant  methods  of  inference  Support  of  multiple  lines  of 
reasoning  is  common  in  the  more  complex  expert  systems  All  of  these  factors  work  to  provide  a 
higher  reliability  measure  for  the  code  providing  the  right  answer  since  it  has  been  tested  with  a 
variety  of  inputs  in  a  variety  of  contexts.  Similarly,  the  data  or  knowledge  has  been  massa  »ed  by 
several  different  processes  so  unexpected  results  become  less  common 

Redundant  knowledge  and  inference  processing  work  together  to  reduce  the  (  fleets  of  mice  taiuty 
and  missing  data.  During  the  development  process,  they  help  to  highlight  inconsistency  With 
proper  development  team  discipline  during  the  period  of  iterative  changes  to  the  KBS  sys  cm.  it 
becomes  robust  in  handling  uncertainty  and  inconsistency  Issues  of  trustworthiness  ma  e  tin- 
aspect  of  redundancy  exploitation  more  important  in  the  case  of  autonomous  systems  than  ones 
with  more  human  interaction  and  control.  Strategies  for  its  employment  in  PM  <'  npph  at  ions 
hinge  on  I  lie  continual  ion  of  research  to  define  conditions  for  t  his  redundancy  and  tin  orpoiai  ion  of 
this  knowledge  in  a  set  of  knowledge  based  tools  used  bv  the  development  teams 

3.4.5  Development  Environment 

Development,  environments  for  KBS  software  have  been  among  t  lie  most  powerful  available  be  ausc 
of  their  emphasis  on  a  user  interface,  debug  facilities  and  tight  coupling  to  the  laugu  ige  of '  hoice 
lOllorts  in  the  last  twenty  years  to  create  a  programming  environment  and  tools  for  the  production 
of  large  and  complex  programs  have  been  successful  in  providing  high  resolution  graphics,  multi- 
windowing  and  an  integrated  environment  to  enhance  user  productivity  by  shortening  the  <  \cle  of 
edit,  compile,  link,  debug,  and  execute.  Increased  productivity  due  to  this  cycle  shortem  tg  ha- 
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3.4.6  Exploratory  Programming  Stylo 

been  accomplished  through  a  reduction  of  operational  steps  in  the  cycle,  and  the  capabeity  to  do 
incren  ental  compiles  and  function  evaluation. 

Tools  developed  include  the*  programming  languages  themselves  such  as  MSP.  I’KOLlbJ  and  OPSfi 
LISP  lias  the  capability  to  create  and  embed  languages  which  have  been  used  for  '.asks  such  as  de¬ 
pendency  analysis,  browsing,  inference,  truth  maintenance,  constraint  propagation  and  knowledge 
representation.  A  class  of  computers  called  LISP  machines  provides  the  power  of  an  integrated 
software  development  environment  by  integrating  these  tools  with  editors,  menu.ng  systems,  and 
the  capabilities  of  the  LISP  language  There  are  some  hardware  enhancement.,  that  support  this 
environment  including  high  resolution  graphics  display,  data  type  tagging,  and  directed  machine 
architectures  Future  capabilities  will  include  parallel  processing  and  memory  features  which  will 
allow  more  powerful  search  and  control  strategies  to  be  executed  within  the  machine  resources 

Special  tools  exist  to  aid  in  the  generation  of  expert  systems.  These  tools,  called  Expert  System 
Shells,  are  available  to  assist  in  the  development  of  many  standard  architectures  for  expert  systems 
These  shells  provide  additional  integrated  environment  services.  Inference  engines,  knowledge  rep¬ 
resentation  schemes  and  graphics  oriented  interfaces  provided  by  the  shells  are  readily  available  for 
install  ition  on  a  wide  variety  of  host,  machines. 

The  commerc  ial  proliferation  of  expert  system  shells  and  the  wide  availability  of  differing  tools 
and  languages  provides  many  choices  in  the  selection  of  an  environment.  Important  considerations 
other  than  cost  include  the  amount,  of  overhead  that  is  allowed,  suitability  of  the  representation 
of  knowledge,  and  inference  techniques  that  are  required  There  have  been  many  observations 
of  an  iterative  cycle  of  tool  building  and  knowledge  acquisition,  which  may  he  shortened  by  the 
appropriate  selection  of  tools  if  enough  is  known  about  the  problem  area  initially 

There  ire  efforts  underway  to  provide  further  power  in  the  integrated  environment  for  the  use  of 
development  teams.  Other  efforts  of  value  to  the  KBS  developers  and  project  personnel  include  the 
development  of  a  knowledge  based  editor  of  which  KBEmacs  is  one  of  the  most  widely  documented 
examples  Tools  to  aid  in  the  development  of  KBS  software,  but  to  be  n-ecl  by  personnel  other 
than  the  Al  development  engineers,  would  include  a  test  management  assistant  and  a  project 
management  assistant  designed  to  import  knowledge  from  the  development  area  anil  aid  the  test 
team  or  the  management  team  in  reaching  decisions  affecting  the  development  and  test  of  the 
system 

Increased  capability  in  the  development  environment  to  obtain  the  best,  performance  from  the 
computer  language,  machine  technology  and  development  team  will  provide  benefits  to  the  HM/C  ' 
development  effort.  in  schedule,  cost  and  performance  risk  reduction.  Of  primary  import  to  the 
success  of  KBS  projects  that  can  have  impact  on  the  IlM/C'  problem  is  the  need  to  identify  and 
develop  tools  dial  can  be  used  to  support  multiple  teams  of  developers.  This  class  of  tools  would 
also  include  some  of  the  tracking  of  progress  necessary  for  the  management  and  test  assistants. 

3.4.0  Exploratory  Projrmmiiiiiijr  Stylo 

Fxplor.it  <>i\  programming  is  I  lie  'conscious  intertw  ining  of  system  design  and  implementation” 
fi|,  Shell i  Iterogniging  that  some  applications  are  design,  rather  than  straight  implementation 
problems,  i  he  not  ion  of  exploratory  programming  allows  t  he  design  to  surface  from  experiment  ing 
with  the  program  In  essetu  the  design  and  the  computer  program  are  developed  together 
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3.4.7  Rapid  Prototyping 


This  technique  is  extremely  valuable  when  dealing  with  large  and  complex  systems  (e  g  /j,\f  (  ) 

for  which  it  is  extremely  difficult  to  postulate  a  complete  specification  Some  reasons  behind  tins 

difficulty  are: 

•  Complexity  itself  -  a  design  engineer  simply  cannot  anticipate  all  of  the  requirements  of  a 
large  scale  system  in  advance  of  the  implementation. 

•  Fluid  requirements  -  for  instance,  during  development,  the  hardware  changes  or  the  particular 
databases  for  which  the  system  is  to  consult  changes. 

•  Human  factors  -  it’s  difficult  to  specify  user  interface  requirements  in  advance  Interfaces 
generally  undergo  extensive  empirical  testing  to  determine  whether  or  not  they  are  effective 
and  considerable  redesign  to  make  them  so 

Regardless  of  the  reason,  a  large  system  with  changing  specifications  leads  to  disaster  when  con¬ 
ventional  software  development  procedures  are  used.  Namely,  the  existing  technology  assumes  that 
the  specification  is  fixed  and  that  the  implementation  conforms  to  the  specification.  Howe’er, 
if  a  major  change  in  the  specification  arises,  every  phase  of  the  project  may  have  to  be  redone 
-  preliminary  design,  detailed  design,  coding,  unit  test  and  integration  Of  course,  the  later  the 
specification  changes  in  the  development  cycle,  the  more  work  that  has  to  be  redone.  In  addition  to 
compromising  the  budget  and  schedule,  the  quality  of  the  end  product  is  also  likely  to  be  affected 
in  the  rush  to  complete  the  system  as  close  to  the  deadline  as  possible. 

With  exploratory  programming,  the  specification  is  expected  to  change  as  a  result  of  (hr  implemen¬ 
tation.  For  example,  suppose  an  initial  system  specification  is  generated.  Dim-  I  lie  sy,(em  begins 
to  be  implemented,  some  of  the  uncertainties  may  be  better  understood,  or  perhaps  concerns  will 
be  uncovered  that  had  not  been  previously  identified.  Consequently,  the  spec  die, it  ion  is  modified 
and  the  exploratory  programming  effort  is  redirected  This  iterative  process  continues  until  at 
some  point,  portions  of  the  system  requirements  begin  to  stabilize.  However,  it  is  important  to 
recognize  that  exploratory  programming  may  not  be  an  end  in  itself  because  of  the  many  rev  isions 
made,  the  code  may  be  inefficient  and  unstructured  when  it  first  achieves  functional  an  eptabilitv 
If  efficiency  and  structure  are  important  issues,  the  rode  can  be  reimplemented  using  traditional 
top-down  techniques.  By  the  time  this  stage  is  reached,  t.he  specification  is  not  likely  to  change  in 
a  major  way  and  the  implementation  is  not  expected  to  be  done  more  than  om  e 

Because  exploratory  programming  calls  for  quick  implementation  of  a  system,  and  i-  g<  in-rallv 
performed  by  a  small  development  team,  computer  systems  conducive  tot  Ins  act  i\  U  y  inn-1  <-i  ham  e 
the  programmer’s  product  ivity  Namely,  programming  tools  such  as  t  lm«e  discussed  in  t  lie  pnvious 
subsection  must  be  available  to  the  development  team 

3.4.7  Rapid  Prototyping 

Rapid  prototyping  involves  a  process  of  recycling  through  system  implement  at  1  u  <uid  testing  m 
order  to  determine  the  validity  of  a  knowledge  representation  scheme  Ks-uii  ial'y .  die  prototype 
a  product  of  this  process,  represents  an  aspei  t(s)  of  the  expert  sysi.-m  ,h-i  -u  1  ,,  ■  „,|  in 

demonstrate  ill  what  manner  encoded  facts,  relationships  and  inference  .-I  rategu  -  I'lhm  ,  In  expert  - 
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3.4.7  Rapid  Prototyping 


knowledge  'Hits  section  outlines  the  cyclic  nature  of  rapid  prototyping  a. id  sm.essiv-  i  <  fii..-mei.l 
in  relat  ionsliip  to  A I  system  development 

The  rapid  prototyping  process  Iwfins  with  acquisition,  com  eptualizai  ion  m  I  thin  f.  *m>nli/nl ion  o! 
knowledge  into  specific  dat  a  st  rnrl  ures,  inference  rules  and  coni  ml  ;,t  rat  gu-s  Tin-  "ex'  s,  r  p  involves 
the  implementation  of  a  demonstrat  inn  prototype  in  order  to  test  the  adequ.u  ,  <>|  I  lie  fi  rmaliratioti 
The  prototype’s  knowledge  base  is  implemented  through  the  aid  of  tools  siHi  n>  intelligent  editors, 
languages  etc  which,  if  n  «?  available,  ;u<  developer!  Perhaps  the  most  import  art  aspect  of  l, lu- 
prototype  lies  in  the  selected  knowledge  representation  scheme 

The  demonst  ra‘  ion  prototypes  purpose  resides  m  proving  the  feasibility  of  ihe  -a  -  Icm  in  ordei  to 
obtain  a  financial  commitment  from  i  he  ii'-er  and  m  testing  lertain  aspect-'  of  prof  lent  definition, 
scoping  and  representation  of  the  domain  Therefore,  heavy  user  involvement  is  necessary  to  the 
financial  success  of  the  system  and  a  strong  domain  expert  commitment  is  required  to  determine 
the  system’s  accurac  v  In  fart,  the  expert  usually  works  along  «viih  the  knowledge  engineer  to 
discuss  problem-solving  strategies  throughout  all  the  development  phases  l  itim  itely,  the  expert 
recognizes  and  points  out  ineffective  problem-solving  techniques  in  the  prototype 

However,  the  success  of  the  prototype  also  depends  on  the  ability  of  the  kriovv|e  >M  engineer  to 
formulate  a  set  of  performance  criterion  with  well-defined  input  and  output  foi  test  measurement 
purposes.  The  idea  is  to  facilitate  an  expert’s  ability  to  recognize  the  prototype’s  efficiency  and  ac¬ 
curacy  particularly  with  regard  to  the  knowledge  base  and  the  inference  struc  ture.  One  knowledge 
engineering  technique  includes  the  use  of  predetermined  case  studies  which  are  initially  reviewed  in 
an  interview  with  the  domain  expert  and  then  compared  with  the  prototype’s  means  of  handling 
the  same  information  The  expert  must  be  aware  of  the  prototype’s  purpose  to  properly  ascertain 
its  functions.  Testing  becomes  more  complex  when  one  considers  that  a  prototype  only  reflects  a 
fraction  of  the  system’s  actual  capabilities.  Frequently,  a  prototype  sacrifices  performance  in  order 
to  concentrate  on  functionality  or  concentrates  on  the  user-interface  and  sets  aside  system  func¬ 
tionality.  file  expert  ’s  involvement  to  test  the  prototype  usually  encourages  more  determination 
to  gather  the  appropriate  knowledge  and  alerts  the  expert  on  how  the  system  functions  in  order  to 
know  how  to  fine  tune  the  actual  product 

A  successive  refinement  of  the  prototype  generally  follows  the  demonstration.  The  prototype  has 
served  a  purpose  I  o  ident  ify  errors  such  as  incorrect  inference  rules,  invalid  knowledge  and  inaccurate 
control  strategies  Another  cycle  through  implementation  and  testing  corrects  the  errors  and  brings 
up  furt fu  r  problematic  areas.  Prototype  development  from  the  initial  demonstration  to  the  mature 
system  seems  to  involve  a  five  step  process:  !67,  Waterman] 
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3.4.8  S nj.iiJ  Development  Teams 


The  demonstration  prototype,  as  previously  described,  attempts  to  prove  the  leasibilny  of  a  system 
for  financial  support  purposes  and  to  test  the  knowledge  representation  and  problem-definition 
techniques.  The  reseairh  prototype  is  generally  a  medium-sized  program  wit  It  powerful  performance 
capabilities,  but  with  a  tendency  to  sillier  from  insufficient  test  mg  I'lie  held  pn >loi  y  pe,  an  average 
to  large-sized  program  also  displays  good  performance.  However,  the  held  prototype  differs  from 
the  research  prototype  in  that  the  system  is  much  more  reliable  because  of  extensive  testing  The 
production  prototype  tends  to  perform  well,  be  reliable  and  fast  Oftentimes,  the  production 
prototype  has  been  reimplement ed  in  a  m<  <>■  efficient  language  Finally,  few  systems  actually  reach 
the  stage  of  a  mature  system  which  is  a  production  model  used  commercially 


3.4.8  Small  Development  Teams 


Although  KBS  systems  constructed  to  date  are  large,  they  have  typically  been  developed  by  sum// 
teams.  In  1982  a  survey  was  conducted  with  the  intent  of  estimating  the  number  of  manyears 
required  to  build  various  Al  systems  [15,  Davis|.  In  the  survey,  ten  systems  were  represented, 
including  Hit  well  known  MACSYMA.  HEARSAY,  DENDKAL.  MYClN.  PUFF,  PROSPECTOR 
and  XCON  systems  One  interesting  result  of  the  survey  is  that  every  system  included  was  devel¬ 
oped  by  an  average  full-time  manpower  effort  of  two  to  five  people 

In  conventional  software  systems  where  the  code  is  generally  modular  and  functions  are  loosely 
coupled,  the  project,  can  logically  support  a  large  staff  Namely,  one  or  more  people  can  be  assigned 
to  design,  code  and  test  ea<  h  major  module  This  concept  cannot  be  extended  to  KBS  systems 
when1  development  does  not  follow  top-down,  modular  techniques 

The  development  of  expert  systems  requires  people  wit  h  a  variety  of  skills  The  following  breakdown 
is  typical  for  a  moder  tely  difficult  problem  domain  [07.  Waterman! 


Domain  expert  (7  V  f  commitment) 


Senior  knowledge  engineer  (2.V7  commitment  ) 


Junior  knowledge  engineer  ( I00c'o  commitment  ) 


Programming  support  ( l(M)°7  commitment) 


The  I  <>l  id  ell  or  I  lor  I  lie  abo\  e  list  mg  is  three  full  - 1  line  professionals  \\  a  I «  nii.1  n  ex  I  end  -  this  i  •  1 1 1  •  j  i 
to  address  mol  e  i  oinplex  o  stems  The  sum  null  \  show  u  below  depi.  Is  I  he  si/c  of  the  'lev  eloj  m<  nl 
team  as  a  film  turn  of  projeel  complexity 


Moderately  dillii  ult  2-1  people. 


Dlllicull  :  3-.r>  people 


Very  dlllicull  l-t>  people 


In  all  cases,  it.  is  cncial  that  a  company  insure  the  accessibility  ml  .,  . 

expert  (s)  The  ex  per  (s)  must  dev  ot  e  up  to  7-V  i  of  1  liei  i  lime  u.  the  b  'I'limi 
and  up  to  half-t  ime  i  hereafter  to  ensmi  the  possibility  of  a  sin  a  esslnl  i  i  <  •  i« 
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3.4.9  User  Involvement 

The  knowledge  engineers  on  the  project  extract  information  from  the  domain  expen  (s)  and  encode 
(his  information  into  the  chosen  method  of  knowledge  representation  Tin  knowledge  engineers 
may  actualb  construe*  the  underlying  framework  in  which  to  encode  the  domain  knowledge  or 
purchase  an  off-the-shelf  development  tool  to  do  so  (Section  3  4.5) 

Programming  support  can  he  used  to  aid  the  KBS  and  or  conventional  prog  imnung  ellorl ,  de¬ 
pending  on  the  project  needs  Conventional  programming  may  he  needed  to  implement  the  more  al¬ 
gorithmic  portions  of  the  system,  or  to  integrate  the  system  with  existing  software  ste  ft  ns  dal  abases 
or  sensor  input s. 

Small  development  teams  constructing  large  systems  imply  high  productivity  w!  ■;«  team  members 
often  work  at  or  above  their  ability  to  manage  system  complexities  Nonetheless,  die  development 
process  is  strongly  influenced  by  the  availability  of  powerful  software  development  tools  (Se<tiou 
3.4.5)  that  help  to  manage  this  complexity. 

The  small  development  team  concept  also  seems  to  preclude  the  need  for  frequent  documentation 
since  information  can  he  verbally  disseminated  with  relative  ease.  Namely,  changes  made  to  a 
program  can  be  verbally  communicated  to  team  members  eliminating  the  n°vd  to  generate  docu¬ 
mentation  such  as  Engineering  Change  Orders  (ECO's),  revisions  to  previous  system  documentation 
or  even  internal  memos 

The  majority  of  t  he  case  st  tidy  evaluations  shown  in  Section  4  indicate  development  teams  comprised 
of  less  than  0  people.  'Phis  observation  is  generally  consistent  with  the  literature  On  the  other 
extreme,  where  system  complexity  and  magnitude  are  very  high,  the  size  of  t  he  development  team 
may  be  much  larger  For  example,  the  team  supporting  the  ongoing  production  of  XCON,  a  DEC 
computer  configuration  system,  numbers  35  The  project  team  is  broken  up  into  four  major  factions: 

•  Administrative  support  (3), 

•  I  ser  support  (5): 

•  Technical  support  (conventional  software)  (6)  and 

•  Knowledge  engineering  ('31! 

More  iotails  pertaining  to  the  XCON  system  are  presented  hi  Section  4  and  Appendix  C.  The 
point  is  that  the  scale  and  complexity  of  SDI  BM/C'  systems  may  be  comparable  to  XCON  If 
this  is  the  case,  const  ruction  <4  systems  in  the  SI)1  context  may  deviate  from  the  concept  of  a  small 
development  team  effort 

3.4.9  l ’s»>r  Involvomonf 

\  user  may  be  smiplv  defined  as  anyone  who  uses  the  KBS  system,  including  the  domain  expert(s), 
knowledge  engmeer(«).  too!  bmlder(s)  or  actual  end-users  The  tool  builders  may  be  involved  in 
debugging  the  system  in  later  stages,  whereas  the  knowledge  engineer  uses  the  tools  available  to 
implement  ind  rriinr  the  system  knowledge  An  end-user  can  be  thought  of  as  the  person(s)  for 
whom  the  system  was  developed  Although  essential,  the  importance  of  end-user  involvement  in 
the  KBS  system  development  process  may  not  be  obvious  Consequently,  it  is  useful  to  consider 
the  stages  of  system  development  m  which  end-users  can  positively  influence  project  evolution: 
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3.4.10  Dnciunrntxtiuii  Produced 


•  System  definition  -  Similar  to  a  conventional  soft*  an  |>r«  *j*-t  ( .  il  is  csMiiiial  that  i  li<  s^(,  m 
meet  the  user’s  needs  Consequently,  user  input  must  be  ■  oiisidered  in  t  In 1  sy '-lein  - j >« ■<  1 1 1  <  iitmi 

phase. 

•  Prototyping  -  As  discussed  in  Section  3  IT,  prototyping  allows  the  users  to  <>bs<..\(.  and 
comment  on  the  evolving  system  A  particularly  important  feature  of  any  system  i<  1  lie  user 
interface,  which  often  comprises  a  lame  segment  of  the  total  system  for  example  m  the 
Dipmeter  Advisor  project,  the  f<  Mowing  breakdown  in  percentage  of  code  was  observed  <>2. 

Smith]: 

-  Inference  Engine  R% 

-  Knowledge  Base  22% 

-  Feature  Detection  1 30f 
Support  Environment  I.V'e 

-  User  Interface  42% 

The  amount  of  effort  expended  on  developing  a  user  interface  for  the  Dipmeter  \<h isor  is 
consistent  with  other  projects  whose  intentions  were  to  facilitate  running  of  the  system  In 
non-sophisticat.cd  users  ]63,58,  Stanley,  Schwartz!  Furthermore,  if  the  system  is  <  umbersome 
to  use  or  if  the  output,  such  as  results  or  error  messages  are  not  clear  and  relevant,  liters  are 
not  apt  to  abandon  existing  manual  methods 

Explanation  facilities,  whereby  a  system  explains  how  solutions  were  reached  are  alsc  a  per¬ 
tinent  part  of  a  user  interface.  A  user  gams  more  confidence  in  the  system  when  he  becomes 
comfortable  with  the  reasoning  process  employed  to  reach  a  roncltisinn(s)  The  explanation 
data  may  also  be  useful  to  the  tool  builder  and  knowledge  engineer  111  debugging  the  cvstem 

•  Testing  -  End-users  and  the  domain  expert  often  play  an  important  role  in  test  mg  by  hypot  h- 
esizing  test  cases  and/or  reviewing  results 

The  majority  of  the  case  study  evaluations  presented  in  Section  1  indicate  that  the  user  <  (immunity 
was  actively  involved  in  system  development  and  that  user  satisfaction  was  often  used  as  a  measure 
of  system  success.  If  the  users  are  an  integral  part  of  the  development  process,  they're  less  likely 
to  feel  threatened  by  the  system  or  experience  the  “not  invented  here''  syndrome  The  result  is 
a  group  of  users  who  look  forward  to  system  completion  in  light  of  utilizing  the  end  prod>:<t  to 
facilitate  l  heir  jobs 

The  enroll  rage  meat  of  user  input  t  hroughout  a  KBS  system  development  process  d  diet's  Iron;  ;  Eat 
of  a  conventional  software  engineering  project  whereby  a  predefined  number  <>f  reviews  me  li<  Id  at 
fixed  points  (Section  3.1  1)  Perhaps  increased  user  participation  in  the  conventional  w  rid  wo.ild 

lead  to  fewer  mismidersl  audmgs  (  mcerumg  system  functionality  and  a  Imther  -urn  ss  ra> .  terms 

of  customer  accept  mice 

3.4. 10  Documentation  Produced 

The  absence  of  an  agreed  upon  model  or  method  for  developing  KBs  sy-iui  1  u  e  r-  -idled  in  the 
lack  of  anv  standard  related  to  the  types  and  kinds  of  documented  111!  amui  1  n  !"•.  cwarv  lor  the 
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3.4.10  Documentation  Produced 


development  ami  support  of  KBS  systems.  In  Building  F.rpi  rt  Sy«f<T>»*!3:>.  Il.iyes-Hotlii.  He-  author 
discusses  problem  identification  during  the  Identification  stage,  defining  key  conn  pis  arc  relations 
during  the  Conceptual  phase  and  developing  partial  specifications  during  the  C  rm  ili/al  mn  phase 
All  of  these  events  have  reference  t<>  information  that  could  well  1.  c.ipinrcd  in  :  he  form  of  a 
document.  For  instance,  there  are  numerous  references  to  providing  users  documentation  on  how 
to  run  a  KBS  system  However,  there  are  no  details  concerning  what  level  of  detail  and  what  kind 
of  information  should  be  included  in  the  documentation  for  the  system  ■’•'>.  Mayes-Hot  h] 

Other  document  requirements  in  the  development  of  KBS  systems  are  di-<  ussed  in  which  the 
develop-  tent  of  a  paper  knowledge  base,  a  knowledge  acquisition  grammar,  inieinal  know  ledge  base 
formats,  knowledge  base,  inference  engines,  and  interface  are  all  presented  as  project  documents 
1 20,  Freilmg  The  donmient  at  ion  scheme  presents  information  that  results  fiom  a  six  step  process 
defined  as 

•  familiarisation: 

•  organizing  knowledge. 

•  representing  knowledge. 

•  acquiring  knowledge; 

•  inference  strategy  design,  and 

•  interface  design. 

On  the  XCON  project  at  DKC,  the  functional  specifications  were  understood  rather  than  written 
because  of  the  ongoing  revision  process  that  occurred  during  the  development  [59,  Scown|.  Design 
specifications  were  embodied  in  the  coded  product  since  expert  systems  do  not  lend  themselves  to 
defined  algorithmic  solutions  where  specifications  can  be  written 

The  need  for  documentation  on  KHS  projects  has,  to  some  degree,  I  een  driven  by  the  nature  of 
KBS  developments  that  have  occurred  to  date  First,  most  KBS  projects  are  small  with  2  to  5 
developers  involved.  Berause  of  the  limited  number  of  development  team  members,  little  need  for 
documentation  exists.  The  iterative  nature  of  KBS  software  development  efforts  also  make  it  very 
difficult  to  determine  what  and  when  to  document.  It  is  clear  that  a  problem  definition  needs  to 
lie  agreed  to  but  that  implementation  should  not  await  complete  definition  The  introduction  of 
an  early  prototype  leads  projects  to  produce  a  working  model  for  the  customer  so  that  feedback 
and  proof  of  concept  can  he  demonstrated  before  a  large  commitment  to  the  project  is  made 
l'lte  /nniluci  tnv  thing  qm.  klg  approach  allows  little  documentation  time  berause  prototyping  is 
oriented  tow  ards  a  product  approach  rather  than  a  process  approach  (as  defined  in  the  conventional 
software  development  world) 

Another  factor  alb  cling  I  In-  tied  for  documentation  is  the  tools  available  to  support  the  devel¬ 
opment  of  KBS  systems  Some  of  the  environments  available  provide  a  level  of  sophistication  not 
generally  seen  in  conventional  software  developments  These  environments  include  various  graph¬ 
ical  and  windowing  tools  which  allow  users  to  do  development  in  an  environment  which  does  not 
rely  upon  extensive  paper  documentation. 


3.4.11  Toatuif' 


Finally,  the  research  and  development  nature  of  KBS  projects  put  them  in  a  category  where  tech¬ 
nology  feasibility  is  of  the  essence  and  not  the  ultimate  deployment  and  support  of  the  product 
In  these  rases,  emphasis  is  certainly  not  on  documentation  since  management  insight  takes  on  i 
different  flavor  and  users  are  not  expected  to  enter  the  picture  until  some  unleiei  minute  future 
point  in  time. 

3.4.11  Testing 

Testing  is  the  comparison  of  a  system’s  actual  behavior  with  its  inti  behavior 

In  conventional  software  development,  a  developer  is  typically  presented  with  a  system  specifics!  ion 
which  is  a  statement  of  the  problem  to  be  solved  by  the  new  system  and  some  idea  of  what  a  soiut  ion 
to  that  problem  might  be  Implicit  in  such  a  specification  is  a  functional  statement  of  the  systems 
behavior  under  a  number  of  real  or  envisioned  situations.  The  developer  takes  that  soecificat ion 
and  embarks  on  the  software  development  process  involving  several  steps: 


•  Requirements  Definition  a  first  level  decomposition  of  the  problem  and  envisioned  solution 
and  an  allocation  of  requirements  from  the  system  specification  to  the  highest  level  component  s 
of  the  system 

•  High  Level  Design 

•  Detailed  Design. 

•  Coding  and  Debugging  (or  unit  testing)  in  which  program  code  is  developed  and  structural, 
as  opposed  to  behavioral,  errors  are  detected  and  removed. 

•  Integration. 


Following  the  integration  step,  a  completed  but  unproven  software  system  exists  This  system 
should  exhibit  the  developer’s  interpretation  of  the  system  behavior  defined  in  the  specification 
There  is  no  assurance,  of  course,  that  it  is  the  problem  solution  suggested  In  the  specification 

In  parallel  with  the  development  team,  the  test  team  translates  the  customer  specification  into  test 
procedures  that  model  the  system  behavior  in  terms  of  stimulus, response  pairs  (as  in  “Push  the 
button;  the  light  turns  green”  or  “Filter  the  file  name;  the  system  responds  no  such  file  This 
process  involves 


•  Hi  quin  men! «  Definition  in  which  the  specification  is  examined  for  essential  requirements 
whether  explicitv  stated,  implied,  or  derived  during  the  design  process 
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3.4. 11  Testing 


Because  (lie  conventional  software  model  usually  requires  that  the  test  procedures  be  approved  by 
the  customer  prior  to  the  initiation  of  any  formal  validation/testing,  the  procedures  are  considered 
a  correct  interpretation  of  the  specification.  The  test  process  consists  of  comparing  (  lie  software  as 
built  with  this  “right  answer.” 

The  ability  to  specify  functional  behavior,  as  is  done  during  conventional  software  development, 
implies  that  the  problem  to  be  solved  is  well  enough  understood  so  that  executional  details  of  the 
resulting  solution  are  known.  Specifications  do  exist  in  the  KBS  world,  but  they  tend  to  be  problem 
statements  without  much  of  a  hint  as  to  what  an  acceptable  solution  might  be. 

As  they  proceed  from  the  problem  statement,  KBS  software  developers  often  enlist  the  aid  of 
“experts"  to  guide  them  in  the  direction  of  an  acceptable  solution  This  permits  the  generation 
of  a  system  which  displays  some  approximation  of  this  expert's  idea  of  solut  ion  behavior.  Because 
they  desire  independence  from  the  development  team  and  because  the  number  of  experts  is  limited 
(frequently  to  a  single  individual),  a  similar  approach  in  the  preparation  of  test  procedures  is 
available  to  the  test  team.  Indeed,  even  if  a  different  expert  is  available  to  assist  in  the  development 
of  the  test  process,  the  potential  for  disagreement  with  the  other  expert  exists.  The  question  then 
becomes  one  of  determining  the  “right  answer.”  In  this  case,  unlike  that  of  conventional  software 
development,  the  customer  is  a  doubtful  arbitor  because  of  the  fuzziness  of  the  original  problem. 

Even  presuming  that  some  approach  toward  generation  of  agreed-to  test  procedures  can  be  found, 
additional  problems  plague  those  charged  with  testing  the  KBS  system: 


•  Does  the  system  rorrtrlly  reflect  the  knowledge  given  it? 

•  Is  the  knowledge  given  the  system  correct?  Who  decides?  Systems  can  be  envisioned  where 
this  may  not  matter,  at  least  not  in  the  same  way  as  conventional  software,  since  the  product 
may  be  self-correcting1.  In  this  case,  the  question  may  become  “When  the  system  discovers 
that  it  is  incorrect,  does  it  move  toward  correctness  by  an  acceptable  amount?”  Again,  who 

decides9 


Just  as  A I  sol  tware  is  urged  towards  solution  behavior  by  prototyping,  the  testing  of  AI  software  for 
battle  management  should  be  prototyped.  Examples  where  this  could  be  applied  include  DARPA 
Stal egir  Computing  Programs  for  AIRLAND  2000  and  the  NAVY  FRESH  program.  Additionally, 
upgrades  to  the  PATRIOT  and  AEGIS  systems  incorporate  AI  technology.  Experiments  with  their 
test  programs  could  be  undertaken. 

Development  of  a  lest  advisor  knowledge  based  system  would  enable  the  test  community  to  analyze 
the  test  requirements  for  systems  and  the  results  of  tests  more  efficiently.  This  would  allow  fuller 
test  team  participation  from  project  inception  and  give  the  benefit  of  a  learning  curve  that  included 
the  prototyping  phase  of  system  development.  Such  an  innovation  might  then  permit  combining 
the  fairly  common  “build  a  little,  test  a  little”  AI  philosophy  with  a  new  “test  a  little,  build  a 
little .”  This  latter,  implying  examination  of  evolving  requirements  and  partial  systems,  is  as  much 
a  driver  of  t  he  system  result  as  the  evolving  system  is  of  the  test  process.  There  is  a  synergy  in  this 
effort  to  define  goals  and  criteria  for  test,  tolerance  and  performance.  As  both  lest  and  development 
teams  identify  and  commit  to  these,  the  problem  to  be  solved  becomes  less  certain  This  strategy 
would  correlate  well  with  a  spiral  approach  aimed  at  minimizing  system  development  risks 


<  ni re nt  literature  amt  Sander’s  survey  results  indicates  that  self-correcting  software  is  not  a 
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I  3.4.12  Management  Control  Mechanisms 

i  KBS  projects  are  typically  accomplished  by  small  development  teams  and  therefore  do  not  require 

!  the  extensive  management  control  mechanisms  that  are  usually  necessary  lor  large  scale  l)()|) 

1  system  programs.  Most  AI  research  and  development  projects  are  developed  by  a  single  team  with 

1  very  little  interface  necessary  to  other  organizations  or  systems  Their  control  usually  consists 

j  of  periodic  technology  reviews  of  actual  versus  planned  progress.  The  review  process  is  generally 

’  informal  with  no  predetermination  of  what  information  will  be  discussed.  This  is  m  contrast  to 

the  typical  large  scale  DOL)  software  development  program  which  has  planned  management  and 
product  reviews  at  various  stages  during  the  development  process.  These  reviews  are  often  extensive 
requiring  a  large  resource  committment 

The  management  techniques  employed  for  KBS  systems  is  complicated  by  the  iterative  nature  of 
KBS  development.  Mastery  of  the  traditional  software  development  model  has  hardly  been  attained 
and  KBS  systems  enter  the  scene  to  present  a  whole  new  problem  set  for  modeling.  It  is  clear  that 
few  managers  even  understand  the  world  of  Al  let  alone  know  how  to  manage  the  highly  complex 
iterative  process.  Considering  the  difficulty  of  scoping  a  domain  within  an  expert  system  and  the 
definition  and  building  of  small  increments,  it  is  difficult  to  conceive  of  a  means  for  managing 
and  controlling  a  total  KBS  development  project  from  requirements  to  product  deployment  and 
retirement.  For  that  reason  most  KBS  developments  are  actually  level-of-effbr(  developments.  That 
is,  only  near-term  schedule,  cost  and  performance  objectives  are  defined  Progress  is  then  measured 
in  the  near-term  and  not  over  a  long  span  of  time.  Since  there  are  no  proven  software  cost  models 
for  estimating  KBS  software  cost  and  schedules,  predicting  the  future  is  even  more  complicated 
The  lack  of  a  broad  historical  data  base  of  KBS  software  costs  and  schedule  also  oilers  no  solni  ion 
or  aid  in  predicting  costs  and  schedules. 

Within  the  AI  system  application  area,  there  have  been  few  systems  that  have  been  developed  under 
the  constraint  of  schedules  and  costs  normally  associated  with  DOD  systems.  KBS  acquisitions, 
which  call  for  the  development  of  systems  for  delivery  to  the  DOD  within  a  given  timeframe,  are 
just  beginning  to  appear  within  DOD  These  procurements  are  normally  of  a  standalone  expert 
system  which  has  little  or  no  interface  with  other  systems.  It  is  also  noteworthy  that  t  hose  sy  stems 
are  not  real-time  to  the  same  degree  as  an  embedded  Battle  Management  Command  and  (  ontrol 
System  must  be. 

As  KBS  system  applications  grow  in  size,  there  will  be  an  increased  need  to  divide  the  problem 
solution  into  manageable  modules  in  order  to  develop  AI  systems  under  dictated  schedules  mid 
costs  It  has  been  suggested  that  benefits  be  derived  m  the  maintainability,  testability  and  speed 
areas  by  applying  modularization  [17,  Orcitich).  Management  of  large  projects  can  potentially 
benefit  from  a  modularization  approach  by  providing  meaningful  subsets  of  a  system  on  which  a 
given  team  can  work  on.  One  of  the  experiences  from  a  project  the  size  of  .V'ON  is  that  bringing 
someone  new  onto  a  large  project,  can  take  a  significant  amount  of  tune  to  become  familiar  with 
the  design  Modularization  should  allow  individuals  to  become  productive  sooner 

The  key  to  managing  a  KBS  project  or  program  is  adequate  insight  and  visibility  iut<  tin  !<  w  I- 
opmciit  process  in  order  to  assess  the  status  and  to  predict  the  future  of  the  program  at  any  er-eu 
point  in  time  The  evolutionary  nature  of  the  KBS  development  process  only  further  complicate-, 
the  management  job  As  KBS  applications  continue  to  expand  and  increase  m  m/c.  inanagemen' 
will  be  faced  with  further  challenges  in  managing  and  controlling  the  development  pro.  t" 
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Problem  Statement.  _ J 

1  The  rapid  prototyping  method  docs  not  offer  dear  guidance  . 
on  how  to  produce  a  well-engineered  commercial  product 

2.  Little  knowledge  exists  on  how  to  manage  the  transfer  of  a 
progressive  and  evolutionary  technology 

3.  There  only  exists  a  limited  number  of  engineers  in  A I  software 
development 

1.  There  is  a  need  for  multiple,  specialized  representatives  in 
the  field  of  Al. 

5.  The  knowledge  is  often  ill-specified  because  the  expert  ca/iw.' 
always  express  exactly  what  he/she  knows  about  the  domain 

(’>  With  rapid  prototyping,  the  system  is  always  in  a  slate  of 
flux. 

7.  There  exists  a  lack  of  knowledge  regarding  testing  of  AI  sys¬ 
tems. 

8.  There  often  exists  voluminous  amounts  of  information  in  the 
definition  of  an  expert  system. 


3.5  Difficulties  in  Developing  AI  Systems 


The  following  section  is  concerned  with  identifying  AI  software  problems  cited  from  a  literature 
study  undertaken  at  Sanders  Associates.  Inc.  and  the  case  study  responses  received  from  the 
Sanders  A I  Questionnaire.  A  comparison  of  AI  and  conventional  software  development  problems  is 
made.  The  literature  study  examined  major  problems  that  appeared  most  frequently  in  different  AI 
software  subcategories  which  are  defined  and  outlined  in  Appendix  A.  The  case  study  problems  deal 
with  those  stipulated  in  one  or  more  responses.  The  comparison  of  AI  and  conventional  software 
development  problems  compares  and  contrasts  common  software  problems  and  issues. 


3.5.1  AI  Software  Development  Problems  Cited  in  the  Literature 

('ertam  problems  in  AI  software  development  were  cited  in  more  literature  sources  than  others. 
I  able  5  1  2  lists  the  eight  problems  which  appeared  with  the  most  frequency.  (See  Appendix  A 
for  a  complete  examination  of  all  the  software  development  problems  identified  in  the  literature 
sources)  l  lic  remainder  of  this  section  examines  each  of  the  frequently  identified  problems  in  more 
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3.5.2  Al  SV.'f'fw Proldoins  Observed  front  the  (Vise  Studies 

AI  software  development  is  a  relatively  young  field  ii:  comparison  w  it  li  conventional  si  ill  ware  de¬ 
velopments.  Therefore,  difficulties  w  ith  identifying  ami  ,!•>•' erneiung  the  best  means  to  develop  \  I 
systems  has  arisen.  The  problem  is  compounded  by  the  fact  that  one  of  (lie  most  to  deft  .<<  <  <  (ited 
methods  for  AI  software  development  resides  in  the  use  of  rapid  prototyping  which  oilers  im  i  ienr 
guidelines  on  how  to  produce  a  well-engineered  commercial  product. 

The  lack  of  well-defined  software  standards  end  methodologies  fillers  snto  the  proper  management 
of  AI  software  development.  AI  manager.,  are  confronted  with  a  progressive  and  ov oliit  iorian 
technology  which  by  definition  entails  little  guidance  on  how  to  properly  manage  people  and  the 
efTort. 

A  by-product  of  a  new  technology  can  be  found  in  the  limited  number  of  experienced  personnel 
AI  software  development  is  hurt  by  both  the  lack  of  available  and,  or  experienced  engineers  nr 
managers.  In  order  to  develop  standards  and  methodologies.  AI  needs  specialized  personnel  to 
determine  an  appre  priate  set  of  standards  to  effectively  manage  ami  guide  an  A I  software  effort 

Knowledgeable  engineers  could  help  improve  the  knowledge  acquisition  process,  the  mam  bottleneck 
in  Al  software  development,  by  providing  effective  techniques  to  communicate  with  the  domain 
expert  The  ,roe>  ss  of  extracting  knowledge  for  incorporation  into  a  knowledge  base  is  time- 
consuming  and  difficult .  Adequate  gathering  of  information  depends  on  the  knowledge  engineer's 
skill  at  extracting  information  from  the  domain  expert  who  oftentimes  lue  difficulty  expressing 
what  he/she  knows  about  the  domain 

The  success  of  an  Al  system  depends  on  the  ability  of  the  knowledge  engineer  to  formulate  a  set  of 
performance  criterion  with  well-defined  inputs  and  outputs  for  test  measurement  purposes  With 
rapid  prototyping,  the  idea  is  to  improve  an  expert’s  ability  to  recognize  a  prototype’s  accuracy 
particularly  with  respect  to  the  knowledge  base  and  the  inference  structure  Sime  the  prototype 
only  reflects  a  small  fraction  of  the  system’s  capabilities,  and  uncertainty  is  a  major  factor  in  the 
selection  of  a  solution  approach,  testing  becomes  more  complex  Also  the  constantly  changing 
requirements  introduced  by  a  system  which  is  in  a  constant  state  of  tlux,  contribute  t.->  the  inability 
to  effectively  test  an  AI  system  Once  again,  the  process  involved  in  testing  Al  systems  and  the 
lack  of  knowledge  regarding  testing  of  AI  systems  seriously  affects  the  Al  software  lev clo;>ment 
effort. 

The  last  major  Al  problem  identified  concerns  the  need  to  delimit  the  i  ask  oj  a  problem  whtv 
building  an  expert  system  Knowledge-intensive  problems  may  not  be  clearly  bounded  Cities-  the 
abundance  of  problem-solving  information  is  bounded,  the  role  the  expert  s;,  .••cm  w  d!  pi  y  may 
not  be  sufficiently  defined  to  produce  a  well-defined  and  effective  sy.-.lem 

3.5.2  AI  Software  Problems  Observed  from  the  Case  Studies 

Mnuv  common  A I  software  development  problems  ran  be  identifier!  between  On-  <  •  -  ■  ■  -tudu-  ea'li- 
ered  from  the  Al  questionnaire  (see  Appendix  It)  'flic  nmsi  frequently  mem  nuied  problem-  wer* 

•  a  lack  of  experience  m  managing  export  ;  > stems. 

•  n  need  to  flee  know  ledge  engineers  Irom  t  asks  which  take  ..way  i">n  •  ;•  '"’ft' 

»  a  lar  k  ol  domain  expert  commitment  whir'll  is  vital  In  yoi'  'ii  >'i<  ■  ami 
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£.5.2  AI  Software  Problems  Observed  from  the  Case  Studies 


•  the  need  for  better  testing  methods. 

However,  a  variety  of  other  significant  problems  also  surfaced  with  regard  to: 

•  the  need  to  develop  a  prototype; 

•  the  ability  for  a  system  to  support  the  views  of  various  experts. 

•  the  need  to  allow  for  requirement  changes; 

•  the  need  for  continual  financial  support  throughout  the  Al  system  development  effort; 

•  document  at  ion;  and 

•  overhead  introduced  in  real-time  systems  by  shells. 

An  examination  of  these  problems  follows. 

AI  lacks  the  required  historical  background  from  which  an  appropriate  guide  on  how  to  develop  a 
well-engineered  product  can  be  extrapolated.  Consequently,  management  does  not  have  the  guide¬ 
lines  or  the  experience  necessary  to  help  in  the  administrative  process  of  Al  software  development. 
For  example,  the  system’s  success  depends  on  management’s  ability  to  effectively  allocate  and  man¬ 
age  domain  experts  and  knowledge  engineers.  The  need  for  a  domain  expert’s  time  commitment 
cannot,  be  over  emphasized  Appropriate  knowledge  acquisition,  effective  problem-solving  tech¬ 
niques  and  schedules  are  highly  dependent  on  the  expert's  knowledge  and  availability.  Without  a 
committed  effort  by  the  domain  expert,  schedules  are  delayed  and  further  expenses  are  incurred. 
Another  problematic  area  occurs  when  knowledge  engineers  are  not  provided  with  management’s 
support  to  spend  the  necessary  time  and  effort  to  design  the  most  appropriate  knowledge  represen¬ 
tation  and  inference  mechanism.  Some  suggestions  have  been  to  encourage  the  knowledge  engineer’s 
creative  mind  during  the  prototype  phase  by  freeing  him/her  from  adminiscrative  duties. 

Another  problem  area  for  management  involves  adequate  testing  ar.d  maintenance  of  the  AI  system. 
I  nfortunately,  AI  software  tends  to  be  more  difficult  to  test  because  of  the  complexity  involved 
in  demonstrating  the  feasibility  of  concepts  with  regard  to  requirements.  Generally,  because  of 
the  inexact  nature  of  AI  systems,  multiple  or  even  thousands  of  test  results  could  be  the  correct 
responses  to  one  requirement,  and  thousands  more  may  exist.  Consequently,  the  need  for  a  more 
appropriate  means  to  test  AI  software  has  been  recognized.  With  regard  to  maintenance,  the  need 
to  modularize  the  system  has  arisen  for  larger  systems  to  ease  the  maintenance  task. 

Other  problems  associated  with  AI  software  development  concern  user  commitment  to  the  Al 
project,  system  modifications  to  accomodate  multiple  expert  ideas,  the  overhead  introduced  by 
shells,  and  standards  needed  for  documentation.  The  user's  financial  commitment  to  an  AI  software 
development  effort  is  highly  dependent  on  the  development  and  demonstration  of  a  prototype.  Lack 
of  a  user's  commitment  to  a  specification  and  architecture  for  systems  with  embedded  AI  software 
has  been  identified  as  a  problem  'I’he  need  to  allow  several  experts  to  examine  the  prototype 
oftentimes  results  in  a  modified  or  adapted  user-interface  A  third  problem  involves  the  overhead 
introduced  by  shells  in  real-time  systems.  And  finally,  a  discrepancy  exists  on  how  to  document, 
what  to  include  in  documents,  and  whether  documentation  is  even  necessary. 
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3.5.3  A  Comparison  of  AI  and  Conventional  Software  Development  Problems 


A  general  concensus  among  the  responses  obtained  from  tie-  cum-  studies  is  dial  problems  were 
significantly  reduced  by  the  flexibility  lliat  rapid  pr- totyping  and  success  refinement  provides 
Changes  to  the  requirements,  a  natural  arid  expect*-!  iccurance  with  AI  software  development, 
were  facilitated  through  this  process 


3.5.3  A  Comparison  of  AI  and  <  'onvontional  Software  Development  Problems 

Many  similarities  and  differences  can  be  found  m  a  comparision  of  conventional  and  A I  software 
development  problems,  'flic  following  section  attempts  to  compare  and  contrast  some  of  the  most 
apparent  problems.  Four  major  software  categories,  based  on  a  report  issued  by  the  Department  of 
Defense  (DOD)  Joint  Services  Task  Force  on  Software  Problems,  Hi  port  of  tin  HOP  /usd  /,i;ci  on 
Software  Problems,  [17,  DrufTelj  shall  be  examined  Software  Fife  Cycle,  (environment.  Software 
Product  and  P<  ople. 

A  brief  summary  of  problems  common  to  both  AI  and  conventional  software-  development  follows 


•  ineffective  communication  between  the  users  of  the  system  and  managers; 

•  lack  of  skilled  software  development  managers  and  engineers; 

•  methodologies  are  in  a  state  of  transition: 

•  acquisition  of  tools  is  not  fully  defined; 

•  difficulty  determining  amount  of  testing  appropriate  to  each  life*  cycle  phase. 

•  lack  of  adequate  Quality  Assurance  engineering  practices 

•  software  is  a  labor-intensive  technology. 

•  software  is  reinvented. 

•  user  sal  isfact  ion, 

•  lack  of  Well-defined  metrics. 


shortage  of  skilled  system  engineers  software  engineers  and  managers  and 
problems  with  users  who  do  not  ha\ ••  ( iu-ugh  time  t<>  sp<  u  !  .  m  tic-  pi . a 


Kadi  of  these  problems,  which  arc  discussed  in  subsequent  paracr-ipu-  "it’.-i.  Ilf  -n  -1  fiwan 
categories,  has  a  significant  impact  on  t  lie  software  development  pp-<  - 


3.5.3  A  Comparison  of  AJ  and  Conventional  Software  Development  Problems 


3.5.3. 1  Software  Life  Cycle 

The  Software  Life  Cycle  refers  to  the  relationship  between  (Liferent.  phas«s  of  the  software  de- 
velopn  ent  process  from  initial  requirements  definition  and  analysis  to  sysitm  deployment  and 
maintenance  The  main  components  of  the  Life  Cycle  category  are  Kcquirem  nt s,  Management, 
Acquisition,  Product  Assurance  and  Transition.  Interestingly,  many  of  (he  problems  that  surface 
at  the  requirements  phase,  such  as  cost  and  schedule,  documentation  communication  and  changes 
to  the  software  are  also  characteristic  of  other  life  cycle  phases,  in  particular  management  and 
acquisition.  A  brief  comparison  between  conventional  and  Ai  life  cycle  problems  follows. 

As  previously  indicated,  some  Requirement  problems  involve  changes  in  software,  cost,  and  sched¬ 
ule,  documentation,  and  communication  The  affect  that,  requirements  changes  have  on  system 
development  due  to  vague,  incorrect,  or  missing  requirements  differs  between  conventional  and  A I 
software  In  conventional  software  development,  requirements  are  frozen  at  some  predetermined 
point  and  any  changes  involve  a  formal  process  which  may  or  may  not  permit,  the  change.  In  fact, 
a  minor  requirements  change  could  result  not  only  in  a  large  software  effort  to  implement  the  new 
requirement,  but  could  also  introduce  additional  expenditures  and  schedule  delays.  These  cost 
incurrmeuts  are  even  more  significant  when  one  considers  that  oftentimes  requirements  are  defined 
according  to  available  time  and  money  In  contrast,  Artificial  Intelligence  systems  are  developed 
with  the  underlying  assumption  that  requirement  changes  will  be  incorporated  as  new  information 
is  gathered  from  prototyping.  Al  systems  generally  are  developed  in  incremental  stages  of  defining 
requirements,  developing  a  prototype,  and  redefining  new  requirements  based  on  the  additional 
information  obtained  through  research  or  from  the  prototype.  Therefore,  requirements  can  be  ex¬ 
ported  to  evolve.  In  fact,  a  major  goal  of  Al  systems  is  to  encourage  and  expect  the  system  to 
change 

Effect  i  ve  rnmmimicat  ion  between  the  users  of  a  system  and  engineers  is  another  major  requirements 
problem.  In  general,  an  engineer’s  comprehension  of  a  user’s  needs  is  reflected  in  the  ability  to  ade¬ 
quately  document  the  information  Although  conventional  and  A I  software  are  similar  with  respect 
to  communication  problems,  Al  is  affected  differently  by  lack  of  communication.  For  conventional 
software,  an  understanding  of  the  product  being  developed  is  essential  to  the  system’s  success.  In 
contrast,  the  basis  for  many  A I  systems,  in  particular  Expert  Systems,  is  an  expert’s  knowledge. 
Therefore,  the  knowledge  engineer  has  to  communicate  well  with  the  user  as  well  as  understand 
the  expertise. 

Management  of  software  encompasses  various  aspects  of  engineering:  managing  of  software,  people, 
and  skills;  availability  of  tools,  metrics  and  models.  A  characteristic  of  both  conventional  and  Al 
software  development  teams  is  the  lack  of  skilled  project  managers.  Al  development  is  particularly 
affected  because  managers  with  or  without  A!  development  experience  have  no  clear  guides  on  how 
to  produce  a  well-engineered  product. 

In  both  convent  ionai  arid  A I  software  development,  the  acquisition  of  software  and  tools  is  not  fully 
defined  One  problem  is  that  conventional  software  developments  are  not  always  carefully  tracked. 
W  it h  prototyping.  Al  systems  are  always  in  a  state  of  flux  which  makes  configuration  management, 
Millie  ult  T...iN  are  mil  necessarily  a  required  deliverable,  nor  are  they  budgeted  for  in  conventional 
siiltw.iic  In  Al.  Imls  aic  uni  necessarily  identified  before  implement  at  ion 

Similar  problems  in  Product  Assurance  can  be  found  in  conventional  and  A I  soft  ware  development. 
Difficulties  arise  in  determining  the*  amount  of  testing  appropriate  to  each  life  cycle  phase  and 
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3.5.3  A  Comparison  of  AI  and  Conventional  Software  Development  Problems 


defining  an  adequate  means  to  test  requirements.  However,  unlike  A!  systems,  conventional  software 
can  generally  be  tested  except  where  requirements  are  vague  or  incomplete  In  contrast,  oftentimes 
no  definite  method  to  check  whether  A I  requirement  conditions  are  met  exist  s 

Presently,  conventional  and  AI  Software  Methodologies  are  in  a  state  of  transition  Conventional 
software  is  concerned  with  upholding  standards  and  conventions  for  requirements,  design  and  cod¬ 
ing,  while  AI  is  in  a  state  of  defining  a  methodology. 

Overall,  many  common  problems  which  deal  with  communication,  requirements  modifications, 
tools,  and  software  development  are  characteristic  of  both  conventional  and  AI  software,  On  the 
other  hand,  each  of  the  problem  categories  vary  based  on  the  software  applicat  ion.  Where  commu¬ 
nication  with  a  user  is  vital  to  conventional  software’s  success  because  requirements  are  frozen  at 
some  point,  AI  software  works  around  the  communication  problem  by  developing  a  prototype  which 
serves  to  further  understand  and  properly  implement  the  requirements  As  a  final  example,  lest 
methods  and  approaches  have  always  been  a  debatable  subject  in  conventional  software  Likewise, 
AI  software  suffers  from  the  same  problem.  However,  AI  differs  in  that  unlike  conventional  softw  are 
where  some  adequate  test  method  is  usually  available,  a  means  to  efficiently  test  A I  software  does 
not  necessarily  exist. 


3.5. 3. 2  Environment 

Environment  encompasses  tools  and  methodologies  involved  in  the  development  and  support  of 
computer  software.  The  five  main  categories  described  are:  Disciplined  Methods,  Labor-intensive. 
Tools,  Reinvention  and  Capital  Investment 

A  characteristic  problem  of  conventional  and  AI  software  is  the  lack  of  adequate  quality  assurance 
engineering  practices  Conventional  software  suffers  from  the  need  for  improved  disciplined  methods 
to  measure  software  quality  particularly  when  requirements  are  not  well-defined  AI  software  differs 
in  that  software  quality  tends  to  be  difficult  to  test  or  measure  at  all  In  fact,  no  true  scale  available 
can  determine  how  high  a  rating  an  AI  system  can  achieve. 

Another  environmental  problem,  the  labor-intensiveness  of  software  technology,  is  apparent  m 
conventional  and  AI  software.  A  need  to  automate  manual  processes  with  an  ultimate  goal  of 
minimizing  manual  processes  has  been  recognized.  A  recommended  means  of  significant  ly  reducing 
an  AI  manual  process  is  to  automate  the  process  of  acquisition,  organization  and  structuring  of 
knowledge. 

Reinvention,  a  problem  in  conventional  software,  refers  to  the  inability  to  reuse  functionally  similar 
software  developed  for  other  systems  resulting  in  higher  development  costs.  AI  software,  built  n: 
incremental  stages  which  introduce  small  extensions,  relies  on  the  concepts  of  reusable  software 
within  a  system.  However,  reusable  software  between  functionally  similar  systems  is  questionable 

Sufficient  capital  investment  in  conventional  and  AI  software  could  significantly  reduce  environ¬ 
ment  problems  such  as  reinvention  and  la<  k  of  or  inadequate  methodologies  and  tools  No  general 
recognition  of  the  importance  of  capital  investment  to  improve  support  environments  and  thus 
reduce  problematic  areas  exists  for  conventional  software.  AI  software  suffers  from  a  general  lack 
of  funding  for  AI  system  development 
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3.5.3  A  Comparison  of  AI  and  Conventional  Software  Development  Problems 


3. 5. 3. 3  Software  Product 

Software  Product  deals  with  the  operational  embedded  computer  software  and  the  materials  nec¬ 
essary  for  life  cvcle  support  such  as:  requirement  and  design  specifications,  source  code,  test  data, 
system  generation  data,  unique  support  tools,  etc.  This  section  compares  and  contrasts  major 
problems  between  conventional  and  AI  software  for  the  following  categories:  Doesn’t  Meet  the 
Need;  Software  Metrics,  and  Design  Attributes.  Presently,  no  common  problems  between  two 
other  categories,  Documentation  and  Immutable  Software,  have  surfaced. 

A  problem  with  software  occurs  when  the  system  docs  not  meet  user  needs.  Conventional  and  A I 
software  depend  on  user  satisfaction  for  a  system's  successful  deployment  However,  eac  h  differ 
ill  the  manner  in  which  users  must  be  satisfied.  Vague  or  incorrect  requirements  ,n  conventional 
software  seriously  affect  the  final  product  Oftentimes,  errors  in  requirements  result  in  an  inade¬ 
quate  system  which  does  not  meet  with  user  specifications.  An  interesting  contrast  is  AI  software, 
which  developed  incrementally,  encourages  and  enforces  regular  meetings  widi  the  user  to  discuss 
and  update  system  requirements.  Therefore,  requirement  errors  are  not  a  serious  hindrance  to 
the  system’s  deployment.  However,  AI  software  appears  to  confront  a  problem  with  the  user’s 
initial  definition  of  the  system  Frequently,  a  user’s  concept  of  the  capabilities  of  an  Af  system 
exceed  present  day  system  development  possibilities.  Conventional  and  AI  software  do  have  one 
common  user  satisfaction  problem.  Both  lack  effective  methods  to  determine  the  system’s  quality 
and  therefore  do  not  have  a  means  to  estimate  or  predict  a  user’s  level  of  satisfaction  with  the  final 
product. 

Software  metrics  attempt  to  provide  analytic  models  and  empirical  data  on  software  to  help  with  the 
selection  of  software  engineering  techniques,  to  estimate  development  resources  and  evaluate  future 
costs  \  problem  with  conventional  and,  in  particular,  AI  software  is  the  lack  of  well-defined  metrics. 
Poorly  defined  metrics  result  from  the  fact  that  various  approaches  to  conventional  software  exist 
and  no  specific  A I  methodology  is  available.  Given  I  he  variety  (or  lark)  of  standard  engineering 
methodologies,  standard  metrics  are  difficult  to  define. 

System  design  should  provide  an  acceptable  programming  solution  to  problems  identified  in  the  re¬ 
quirements  document.  Some  of  the  major  problems  in  conventional  software  design  are  inadequately 
designed  requirements,  incorrect  assumptions  made  by  the  engineer,  and  ambiguous  requirements. 
In  each  of  the  situations,  the  system’s  ability  to  be  modified  becomes  crucial.  Some  of  the  reasons 
for  poorly  designed  software  can  be  attributed  to  lack  of  a  design  methodology  with  a  top-down 
hierarchical  breakdown  of  the  system  and  lack  of  consideration  for  human  engineering  in  the  design 
of  the  system.  The  result  can  be  a  system  that  is  not  necessarily  capable  of  handling  changes  easily 
and  with  minimal  cost  expenditures  and/or  schedule  delays.  The  final  product  essentially  will  not 
meet  with  user  requirements. 

I  n  I  ike  conventional  software,  no  formal  methodology  exists  by  which  AI  systems  can  be  developed. 
No  clear  guidelines  on  how  to  efficiently  produce  a  well-engineered  commercial  product  through 
rapid  prototyping  is  available.  Nonetheless,  many  of  the  decided  problems  in  conventional  software 
design,  which  canno'  easily  handle  requirement  modifications  for  poorly  designed  systems,  do  not 
appear  (or  are  remedied)  in  AI  systems.  As  a  result  of  the  cyclic  nature  of  AI  systems,  where 
software  is  evolved  repeatedly  from  requirements  to  design  to  redefinition  of  requirements,  incorrect, 
system  specifications  are  incorporated  and  updated  continuously.  In  fact,  perhaps  the  main  problem 
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3.5.3  A  Comparison  of  AI  and  (' onventionai  Software  Development  Problems 


with  the  design  of  AI  systems,  when  considering  user  needs,  is  in  defining  the  knowledge  base  w  1 1  i i 
optimal  problem  solving  techniques. 

3.5. 3.4  People 

In  conventional  and  AI  software  development,  two  problematic  areas  have  arisen  as  a  resuli  of  a 
shortage  of  skilled  system  engineers,  software  engineers  and  managers:  the  number  and  availability 
of  skilled  professionals.  Engineers  with  knowledge  in  various  computer-related  fields  as  well  as 
experts  who  can  successfully  lead  software  projects  are  needed  for  conventional  and  AI  software 
development  efforts.  As  a  relatively  young  field,  AI  cannot  provide  a  sufficient  number  of  experi¬ 
enced  managers  and  engineers.  Strong  management  does  not  exist  to  guide  an  AI  project  through 
completion  and  there  are  not  enough  engineers  with  AI  software  development  experience 

Another  conventional  and  AI  software  problem  is  a  user’s  skill  and  availability  to  effectively  com¬ 
municate  a  system’s  requirements.  AI,  in  particular,  is  hurt  by  users  who  trim  knowledge  to  (it  the 
knowledge  structure;  who  are  not  able  to  express  their  knowledge;  and  who  do  not  have  enough 
time  to  devote  to  the  project. 
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SECTION  4 


Case  Study  Results 


4.1  Case  Study  Data 

Tabular  data  pertaining  to  the  case  study  evaluations  are  presented  in  this  section  for  the  following 
participants: 

1.  ARINC  Research  Corporation; 

2.  Boeing  Computer  Services; 

3.  Boeing  Military  Airplane  Company; 

4.  Brattle  Research  Corporation; 

5.  Carnegie  Group,  Inc.; 

6.  Digital  Equipment  Corporation; 

7.  Expert  Technologies,  Inc.; 

8.  Frey  Associates,  Inc.; 

9.  GTE  Data  Services; 

10.  IBM  Federal  Systems  Group; 

11.  Inference  Corporation  (2  cases); 

12.  Lockheed  Aircraft  Service  Company  (2  cases); 

13  Lockheed-Georgia  Company; 

14.  The  MITRE  Corporation  (Bedford,  Ma  ); 

15.  The  MITRE  Corporation  (McLean,  Va  ); 

16.  Northrop  Avionics  Division; 

17.  PAR  Government  Systems  Corporation  (3  cases); 

18.  Sanders  Associates.  Inc; 

19.  Sclilumberger-Doll  llesearch  (literature  source  only); 

20  Software  Architecture  and  Engineering,  Inc.  (2  cases);  and 


4.1  Caap  Study  Data 


21.  Texas  Instruments,  Inc 

The  tables  present  a  common  set  of  features  with  individual  applications  for  each  participant.  'Die 
manner  in  which  the  data  is  codified  facilitated  the  analytical  process  of  ferreting  out  common 
characteristics,  general  trends  and  distinguishing  traits.  Specific  observations  relating  to  such  as¬ 
pects  are  presented  in  Section  4.2. 

Prior  to  evaluating  the  tabular  data, the  reader  is  encouraged  to  compare  the  cm  in  a  qualitative 
manner  since: 

•  Many  systems  are  in  different  phases  of  development.  Consequently,  the  responses  for  a 
system  in  an  early  stage  of  development  may  change  as  the  system  matures.  For  example,  a 
system  that  is  in  the  phase  of  demonstrating  technology  may  have  had  little  user  involvement. 
However,  once  the  system  has  demonstrated  feasibility  of  the  technology,  users  are  apt  to 
become  more  involved. 

•  Many  of  the  systems  relate  to  different  domains  and  applications.  The  features  pertaining  to 
the  development  of  a  tool  many  be  justifiably  distinct  from  the  features  relating  to  a  complete 
stand-alone  system. 

•  The  concise  responses  required  by  a  tabular  presentation  of  the  data  may,  in  some  cases,  be 
misleading.  To  obtain  a  better  understanding  of  the  systems  studied,  the  reader  is  encouraged 
to  review  the  case  study  summaries  delineated  in  Appendix  C. 
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Table  4.1-1  :  AFtINC  Research  Cor|Miration  -  System  Testability  and  Maintenance  Program 

(STAMP) 
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FEATURE 


Problem  Category 
Domain 

Current  System  Phase 
Type  of  Funding 
Experienced  Al  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
» 'onvenl tonal  S\\  Interlace 
Tools  Used:  Requirements  Analysis 
Tools  Used:  Development 
Tools  Used:  Testing 
Formal  Test  Procedures 
Testing  Criteria 
Test  Data 

Self  Modifying  Code 


APPLICATION 


(leneric  tool 

Electronic  warfare  testability 
Field  support 

Internal  Research  k  Development. 


Very  active 
Not  applicable 
Yes,  internal  standards 
Rules  and  facts 

Information  theory  and  dependency  analy  sis  algorithms 
Yes,  for  the  rehosted  version 
Both  managerial  and  technical  reviews 
Yes,  5  major  architectural  changes 


6  months 

2  years  field  model 
Yes,  extensive 

Not  after  initial  rehosted  system 
STAMP  is  written  in  conventional  software  (Fortran  771 

No 

Fortran  77  compiler/debugger,  and  IIP- WOO  operating  system 

No 

Testing:  module,  integration  testing  2  month  user  trial  period 
Not  applicable 
Actual  usage 


4.1  Case  Study  Data 


Table  4.1-2  :  Boeing  Computer  Services  -  Strategic  Force  Management  Decision  Aid 


FEATURE 

APPLICATION 

Problem  Category 

Replanning  decision  aid 

Domain 

Employment  of  strategic  forces 

Current  System  Phase 

Development 

Type  of  Funding 

Internal 

Experienced  Al  Staff 

Ye* 

Size  of  Development  Team 

1 

Level  of  User  Involvement 

High 

Domain  Expert  Role 

Knowledge  acquisition 

Formal  Development  Procedures 

No 

Type  of  Knowledge  Representation 

Frames,  logic  it  rule  based 

Inference  Mechanism 

KEE  supplied 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

None 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

6  Man-months 

Development  Time:  End  Product 

System  not  fielded 

Documentation 

Some  -  informal 

Configuration  Management 

None 

Conventional  SW  Interface 

None 

Tools  Used:  Requirements  Analysis 

None 

Tools  Used:  Development 

KEE,  TI  Explorer 

Tools  Used:  Testing 

None 

Formal  Test  Procedures 

None 

Testing  Criteria 

Expert  acceptance  of  plan 

Test  Data 

Test  scenarios 

Self  Modifying  (’ode 

No 

A  A 
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Table  4.1-3  :  Boeing  Military  Airplane  Company  -  Automatic  Target  Recognition  (ATR)  Program 


FEATURE 

APPLICATION 

Problem  Category 

Interpretation  -  image  understanding 

Domain 

Target  recognition 

Current  System  Phase 

Feasibility  dcmnnslrat ion 

Type  of  Funding 

Internal  Research  A  Development 

Experienced  Al  Staff 

Yes 

Size  of  Development  Team 

Proprietary 

Level  of  User  Involvement 

Influenced  Si  directed  development  efforts 

Domain  Expert  Hole 

Minimal  expert  consultat  ion 

Formal  Development  Procedures 

Vi'S 

Type  of  Knowledge  Representation 

Frame-based  with  rules 

Inference  Mechanism 

Forward  chaining 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

Continuous  internal  review  process 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes,  minor  rule  changes  and  major  user  interface  modifications 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

Proprietary 

Development  Time:  End  Product 

Proprietary 

Documentation 

Mostly  informal  except  for  annual  report  on  slate  of  research 

Configuration  Management 

Entity  checkpointing  and  file  backup 

Conventional  SW  Interface 

Knowledge  Craft  supports  direct  calls  to  l.isp  code 

Tools  Uscil:  Requirements  Analysis 

No 

Tools  Used:  Development 

Knowledge  Craft  on  Symbolics  processors 

Tools  Used:  Testing 

No 

Formal  Test  Procedures 

No 

Testing  Criteria 

Results  verified  by  user  and  consistent  with  requin  inenis 

Test  Data 

Test  image  inputs  -  also  lired  every  rule 

Self  Modifying  Code 

Not  at  this  time 

i  r. 


4.1  Case  Study  Data 


Table  4  1-4  :  Brattle  Research  Corporation  -  Text  Interpretation  System 


FEATURE 

APPLICATION 

Problem  Category 

Text  interpretation 

Domain 

Business  information 

Current  System  Phase 

Development 

Type  of  Funding 

Venture  and  contract 

Experienced  Al  Staff 

Yes  -  all  members 

Size  of  Development  Team 

12  overall 

Level  of  User  Involvement 

Slight  -  more  later 

Domain  Expert  Role 

Not  applicable 

Formal  Development  Procedures 

Yes,  internal 

Type  of  Knowledge  Representation 

Proprietary  (frame-like) 

Inference  Mechanism 

Inheritance  Si  deduction 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

Yes  -  both  management  Si  technical 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

1  year 

Development  Time:  End  Product 

2  years  (estimate) 

Documentation 

Yes  -  formal  specs  Si  design  documents 

Configuration  Management 

Yes 

Conventional  SW  Interface 

Yes  -  in  development 

Tools  Used:  Requirements  Analysis 

Prototyping 

Tools  Used:  Development 

Symbolics  environment 

Tools  Used:  Testing 

Yes  -  built  own  tools 

Formal  Test  Procedures 

Blind  Si  regression 

Testing  Criteria 

Absolute  accuracy 

Test  Data 

Text  articles 

Self  Modifying  Code 

No 

_ FEATURE _ 

Problem  Category 

Domain 

Current  System  Phase 
Type  of  funding 
Experienced  AI  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
Conventional  SW  Interface 
Tools  Used:  Requirements  Analysis 
Tools  Used:  Development 
Tools  Used:  Testing 
Formal  Test  Procedures 
Testing  Criteria 
Test  Data 

Self  Modifying  Code 


APPLICATION 


Monitor  and  control 
Factory  automated  materials  handling 
Completed  -  production  model 
Commercial  contract 


Present  but  passive 
No  experts  in  this  domain 
No 

Production  rules 


Forward  chaining 


Informal 


Yes,  to  accomodate  customer  requests 
Some  critical  modules 
7  months 
6  months 

Specification  and  informal  memos 


Use  of  mailboxes  for  C  and  BLISS 


('ode  generator  for  database  routines 
Simulator 


Simulator 


FEATURE 

APPLICATION 

Problem  Category 

Planning  and  control 

Domain 

Computer  systems  configuration 

Current  System  Phase 

Production  mode 

Type  of  Funding 

Internal 

Experienced  Al  Staff 

Yes  -  trained  internally 

Size  of  Development  Team 

35 

Level  of  User  Involvement 

Heavy 

Domain  Expert  Role 

Multiple  experts  -  various  roles 

Formal  Development  Procedures 

Yes 

Type  of  Knowledge  Representation 

Rules 

Inference  Mechanism 

Forward  chaining  (OPS5  interpreter) 

Requirements  Analysis 

Yes  -  problem  definition 

Approval  Mechanisms 

Informal  -  management  team  concept 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

For  major  mods  only 

Development  Time:  Initial  Prototype 

5  months 

Development  Time:  End  Product 

12-18  mo.  for  1st  installation/quarterly  upgrades  now 

Documentation 

Architectural  level,  comments  in  code 

Configuration  Management 

Yes 

Conventional  SW  Interface 

Yes 

Tools  Used:  Requirements  Analysis 

No 

Tools  Used:  Development 

VAXll/VMS,  0PS5,  DBMS  +•  some  “home-grown” 

Tools  Used:  Testing 

Yes,  for  code  Az  unit  test  phase 

Formal  'lest  Procedures 

Regression  testing/Problem  reporting  system 

Testing  Criteria 

Qualitative  only 

Test  Data 

Customer  orders-  real  and  hypothesized 

Self  Modifying  ( 'ode 

No 

Ov/> 
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Table  4.1-7  :  Expert  Technologies  Inc.  -  FECASYS 


_ FEATURE _ 

Problem  Category 

Domain 

Current  System  Phase 
Type  of  Funding 
Experienced  AI  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
Conventional  SW  Interface 
Tools  Used:  Requirements  Analysis 
Tools  Used.  Development 
Tools  Used:  Testing 
Formal  Test  Procedures 
Testing  Criteria 
Test  Data 

Self  Modifying  Code 


_ APPLICATION _ 

Automatic  pagination 

Yellow  Page  directories 
Delivered 
Internal 
Yes  -  3 
8 

High 

Knowledge  acquisition 

Semantic  network  of  frames 
Heuristic  search  mechanism 
Yes 

Yes,  at  project  manager/senior  engineer  level 
Yes 
Yes 
Yes 

7  man-months 
69  man-months 
Yes 

Yes  -  proprietary 
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Acceptance  Test  Plan  (ATP) 

Simulation 
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Table  4.1-8  :  Frey  Associates,  Inc.  -  THEMIS  Management  Information  System 


FEATURE 


Problem  Category 
Domain 

Current  System  Phase 
Type  of  Funding 
Experienced  Al  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
Conventional  SW  Interface 
Tools  Used:  Requirements  Analysis 
Tools  Used:  Development 
Tools  Used:  Testing 
Formal  Test  Procedures 
Testing  Criteria 
Test  Data 

Self  Modifying  Code 


APPLICATION 


Natural  language  processing 
Database  query 

Completed  -  commercial  system 
Internal 

Yes,  1  out  of  12 
6-12 

Medium-heavy 
Not  applicable 
No 

Rule-based 
Forward  chaining 
Yes 

Informal 

Yes 

Throughout  system  development 
Yes 

6  months  (1  man  year) 

10  man  years  for  final  product 
Yes  -  extensive  (user’s  manuals  etc.) 

Yes,  source  code  and  version  control 
Yes  (Fortran  and  user  interface) 

None 

InterLISP 

None 

Yes,  built-in  test,  regression,  auto,  error  logging 
Internal  testers  and  user  agreement  with  conclusions 
User  defined  or  hypothetical  queries  (correct  and  incorrect) 
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4.1  Case  Study  Dat;i 


Table  4.1  9  :  GTE  Data  Services  -  Central  Olfice  Maintenance  Printout  Analysis  and  Suggest 
System  (COMPASS) 


FEATURE 

APPLICATION 

Problem  Category 

Fault  Diagnosis 

Domain 

Telecommunication  switch  hardware 

Current  System  Phase 

Limited  field  study 

Type  of  Funding 

Internal 

Experienced  Al  Staff 

•) 

Size  of  Development  Team 

2  -  7 

Level  of  User  Involvement 

Small  -  Domain  Expert  was  supervisor  of  end-uses 

Domain  Expert  Role 

1  week  per  month  met  with  knowledge  engineer:- 

Formal  Development  Procedures 

Yes 

Type  of  Knowledge  Representation 

Frames  and  production  rules 

Inference  Mechanism 

KEE  supplied 

Requirements  Analysis 

No 

Approval  Mechanisms 

Informal  -  supervisory  level 

Iterative  Development  Process 

Only  during  early  phases 

Design  Changes 

Some 

Prototypes  Built 

Two 

Development  Time:  Initial  Prototype 

1 00  man-months 

Development  Time:  End  Product 

106  man-months 

Documentation 

Technical  notes  and  Knowledge  Acquisition  Pules  documents 

Configuration  Management 

Implemented  a  software  control  know  ledge  base 

Conventional  SW  Interface 

None 

Tools  Used:  Requirements  Analysis 

None 

'Pools  Used:  Development 

KEE 

'Pools  Used:  Testing 

LISP  procedure  to  do  testing  in  batch  mode 

Formal  'lest  Procedures 

No 

Testing  Criteria 

Independent  experts  evaluation  of  COMPASS  ripes  and  output 

'Pest  Data 

Hardware  error  message  files 

Self  Modifying  Code 

No 
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4.1  Case  Study  Data 


Table  4.1-10  :  IBM  Federal  Systems  Group  -  Fault  Diagnosis  and  Resolution  System  (FDRS) 


FEATURE 


Problem  Category 
Domain 

Current  System  Phase 
Type  of  Funding 
Experienced  AI  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototyp 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
Conventional  SW  Interface 
Tools  Used:  Requirements  Analysis 
Tools  Used:  Development 
'Pools  Used:  Testing 
Formal  Tv  *  Procedures 

1 1  lie. *  i  "  <  i  ia 
'lest  I )ata 

Self  M  odifying  Code 


V  V 


APPLICATION - 


Fault  diagnosis 

Satellite  ground-based  maintenance 
Installation  Sc  test 

Independent  research  and  development 
Yes,  1 


Production  rules 
Forward  and  backward  chaining 


Standard  research  approval  process 


6  Man-months 
20  Man-months 


Informal 


Commercial  Sc  in-house  shell 


Dynamic  evaluation  ot  nile  bases 
Simulation 


4.1  Case  Study  Data 
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Table  11  12  :  Inference  Corp.  -  Medical  Charge  Evaluation  Control  (Medchec) 


FEATURE 

APPLICATION 

Problem  Category 

Fraud  Detection 

Domain 

Medical  Insurance  ( 'bums 

Current  System  Phase 

Development 

Type  of  Funding 

Contract 

Experienced  Al  Stair 

Yes 

Size  of  Development  Team 

:i 

Level  of  User  Involvement 

High,  they  were  the  experts 

Domain  Expert  Role 

Defined  requirements  of  system 

Formal  Development  Procedures 

Yes 

Type  of  Knowledge  Representation 

Frames  and  production  rules 

Inference  Mechanism 

Forward  k  backward  chaining 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

None 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes,  mostly  low  level  ones 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

6  man-months 

Development  Time:  End  Product 

12  man-months 
(7  calendar  months) 

Documentation 

Some 

Configuration  Management 

None 

Conventional  SW  Interface 

Yes 

Tools  Used:  Requirements  Analysis 

None 

Tools  Used:  Development 

ART,  Symbolics 

Tools  Used:  Testing 

None 

Formal  Test  Procedures 

None 

Testing  Criteria 

No  formal  process 

'lest.  Data 

Live  data 

Self  Modifying  Code 

No 

4-14 


illlllgilL.; 


4.1  Case  Study  Data 


Table  4.1-13  :  Lockheed  Aircraft  Service  Company  -  Export  Software  Pricer  (ESP) 


FEATURE 

APPLICATION 

Problem  Category 

Software  costing 

Domain 

Sizing  of  software 

Current  System  Phase 

Complete 

Type  of  Funding 

Internal 

Experienced  AI  Staff 

No 

Size  of  Development  Team 

2 

Level  of  User  Involvement 

Moderate 

Domain  Expert  Role 

No  expert  used 

Formal  Development  Procedures 

Yes,  internal 

Type  of  Knowledge  Representation 

Frame  based 

Inference  Mechanism 

Backward  chaining  provided  by  LES 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

Yes 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

4  man-months 

Development  Time:  End  Product 

12  man-months 

Documentation 

Software  Requirements  Specification 

Configuration  Management 

Yes 

Conventional  SW  Interface 

Yes 

I'oolr*  Used  Requirements  Annhsis 

None 

Tools  Used:  Development 

LES  and  conventional  compilers 

Tools  Used:  Testing 

None 

formal  'lest  l’ro«  edit  res 

None 

Testing  Criteria 

1  /-  80%  of  .actual  size  and  costs 

Test  Data 

Existing  systems  with  known  size  and  costs 

Self  Modifying  Code 

No 

FEATURE 

APPLICATION 

Problem  Category 

Detection  ii  characterization  of  frequency  hopped  signals 

Domain 

Signal  identification 

Current  System  Phase 

Completed  prototype 

Type  of  Funding 

Internal 

Experienced  AI  Staff 

Yes 

Size  of  Development  Team 

1 

bevel  of  User  Involvement 

High 

Domain  Expert  Role 

Small  -  knowledge  acquisition 

F(  rmal  Development  Procedures 

No 

Type  of  Knowledge  Representation 

Temporal  framework 

Inference  Mechanism 

Temporal  logic  and  pattern  matching 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

No 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

9  months 

Development  Time:  End  Product 

System  not  fielded 

Documentation 

Yes 

Configuration  Management 

No 

Conventional  S\V  Interface 

No 

Tools  Used  Requirements  Analysis 

None 

Tools  Used:  Development 

LISP 

Tools  Used:  Testing 

None 

Formal  Test  Procedures 

No 

Testing  Criteria 

Consistency  and  improved  performance 

lest  Data 

Simulation 

Self  Modifying  Code 

No 

4.1  Case  Study  Data 


fable  4.1  -15  :  Lockheod-Cleorgia  Company  -  Pilot's  Associate 


FEATURE 

APPLICATION 

Problem  Category 

Intelligent  assistant 

Domain 

Combat  avionics 

Current  System  Phase 

Analysis 

Type  of  Funding 

Contract  (US  Air  Force) 

Experienced  AI  Staff 

Yes  65%  of  the  development  team 

Size  of  Development  Team 

Over  40 

Level  of  User  Involvement 

Heavy 

Domain  Expert  Role 

Several  experts  involved  in  knowledge  acquisition 

Formal  Development  Procedures 

Yes 

Type  of  Knowledge  Representation 

Varied 

Inference  Mechanism 

Varied 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

Informal  and  some  formal  reviews  scheduled 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

2  years  (estimate) 

Development  Time:  End  Product 

Not  completed 

Documentation 

Yes 

Configuration  Management 

Yes  -  CMS 

Conventional  SW  Interface 

Undetermined 

Tools  Used.  Requirements  Analysis 

None 

Tools  Used:  Development 

ART,  LES,  OPS5 

Tools  Used:  Testing 

None  so  far 

Formal  Test  Procedures 

Yes 

Testing  Criteria 

Satisfaction  of  system  requirements  plus 
Experts  evaluation  of  system  performance 

Test  Data 

Simula!  ion 

Self  Modifying  Code 

No 

£ 


FEATURE  APPLICATION 


Problem  Category 
Domain 

Current  System  Phase 
Type  of  funding 
Experienced  A1  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
Conventional  SW  Interface 
Tools  Used:  Requirements  Analysis 
Tools  Used:  Development 
Tools  Used:  Testing 
Formal  Test  Procedures 
'lest mg  <  ‘riteria 
Test  Data 

Self  Modifying  Code 


Fault  detection /diagnosis 
Space  technology 
Final  prototype  complete 
Contract  (NASA) 

Yes 

2 

Heavy 

Active  in  system  development 
No 

Frames 

Frame-based 

Yes 

Informal  -  domain  expert 
Yes 

Major  throughout  system  development 
Yes 

6  months 
2  years 

Nothing  formal 
No,  did  tape  backups 
Yes 
No 

Yes,  Symbolics  Zetalisp  environment 
No 
No 

User  agreement  with  conclusions 
Live  sensor  data 
No 


4.1  Case  Study  Data 


Table  4.1-17  :  MITRE  Inc.  (McLean)  -  ANALYST 


FEATURE 

APPLICATION 

Problem  Category 

interpretation /assessment 

Domain 

Battle  management 

Current  System  Phase 

Completed  -  production  model 

Type  of  funding 

Sponsored  research  with  follow  on  contract 

Experienced  AI  Staff 

3-4 

Size  of  Development  Team 

5 

Level  of  User  Involvement 

Domain  expert,  review  and  test 

Domain  Expert  Role 

Active  in  development  and  transition  to  field 

Formal  Development  Procedures 

No,  used  rapid  prototyping 

Type  of  Knowledge  Representation 

Frames  and  rules 

Inference  Mechanism 

Frame- based  and  goal  directed 

Requirements  Analysis 

Matching  of  problems  to  AI  features 

Approval  Mechanisms 

Informal;  domain  expert  and  program  manager 

Iterative  Development  Process 

Yes,  adding  knowledge 

Design  Changes 

At  least  one  major  change 

Prototypes  Built 

One 

Development  Time:  Initial  Prototype 

8  months 

Development  Time:  End  Product 

18  months 

Documentation 

Listings,  nothing  formal 

Configuration  Management 

Prior  to  release  build  level 

Conventional  SW  Interface 

No 

Tools  Used:  Requirements  Analysis 

None 

Tools  Used:  Development 

Micro-compiler  for  Lisp  Machine/  built  others 

'Pools  Used:  Testing 

No 

Formal  Test  Procedures 

No 

Testing  t'ritena 

Domain  expert  and  user  satisfaction 

Test  Data 

Simulated  data  stream 

Self  Modifying  t  'ode 

No 

4-19 


4.1  Case  Study  Data 


Table  4.1  18  :  Northrop  Aircraft  Division  -  Expert  System  for  Target  Attack  Sequencing  (ES  I  AS) 


FEATURE 

APPLICATION 

Problem  Category 

Decision  aids 

Domain 

Combat  avionics 

Current  System  Phase 

Completed  feasibility  prototype 

Type  of  Funding 

Internal 

Experienced  A1  Staff 

Yes 

Size  of  Development  Team 

4 

Level  of  User  Involvement 

Active,  throughout  development  and  testing 

Domain  Expert  Role 

Very  active  for  knowledge  acquisition  and  testing 

Formal  Development  Procedures 

Yes,  conventional  framework 

Type  of  Knowledge  Representation 

Production  rules 

Inference  Mechanism 

Forward  <C  backward  chaining 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

Yes,  frequent  reviews  and  several  demos 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

6  months 

Development  Time:  End  Product 

Not  applicable 

Documentation 

Yes,  enforced 

Configuration  Management 

Yes 

Conventional  SW  Interface 

Not  at  this  time 

Tools  Used:  Requirements  Analysis 

None 

loots  Used:  Development 

LISP  workstations  using  Common  Lisp 

Tools  Used:  Testing 

Yes,  built-in  trace  facilities 

Formal  Test  Procedures 

No 

Testing  ( Vitoria 

User  satisfaction /expert  assessment 

Test  Data 

Usage 

Self  Modifying  Code 

No 
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FEATURE 

APPLICATION 

Problem  Category 

Decision  Aid 

Domain 

Cost/benefit  of  tactical  missions 

Current  System  Phase 

Completed  prototype 

Type  of  Funding 

Contract 

Experienced  A1  Staff 

Yes 

Size  of  Development  Team 

3-6  computer  scientists  <U.  in-house  expert  (s) 

Level  of  User  Involvement 

Moderate 

Domain  Expert  Hole 

Define  the  problem,  Knowledge  engineering  and  lest  prototype 

Formal  Development  Procedures 

Yes 

Type  of  Knowledge  Representation 

Inference  network  of  production  rules  with  confidence  factors 

Inference  Mechanism 

Coal-directed 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

Informal  -  project  stair 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

24  man-months  (including  preliminary  investigation  phase) 

Development  Time:  End  Product 

System  not  fielded 

Documentation 

Yes,  extensive 

Configuration  Management 

Yrs 

*  'nm  eul  tonal  S\\  litlri  lace 

\cs,  support  env  irounn  ut 

Tools  Used  Requirements  Analysis 

None 

Tools  Used:  Development 

In-house  developed  Expert  System  Shell  and  graphics  tools 

Tools  Used:  Testing 

No 

Formal  Test  Procedures 

Yes 

Testing  Criteria 

F.xperts  and  potential  users  ratings  of  system 

Test  Data 

Scenario 

Self  Modifying  Code 

No 

4.1  C'hho  Sourly  Data 
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Table  II  20  PAR  Government  Systems  Corporation  -  Duplex  Army  Radio/ Radar  Targeting 
Decision  Aid  (DARI  ) 


FEATURE 

APPLICATION 

Problem  Category 

Decision  Aid 

Domain 

Target  identificat  ion/classification 

Current  System  Phase 

Completed  prototype 

Type  of  Funding 

Contract 

Experienced  Al  Staff 

Yes 

Size  of  Development  Team 

3-0  computer  scientists  &  in-house  expcrt(s) 

Level  of  User  Involvement 

Moderate 

1  >omain  Expert  Role 

Define  the  problem,  knowledge  engineering  and  test  prototype 

Formal  Development  Procedures 

Yes 

Type  of  Knowledge  Representation 

Inference  network  of  production  rules  with  confidence  factors 

Inference  Mechanism 

Goal-directed 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

Informal  -  project  staff 

Iterative  Development  Process 

Yes 

Design  Changes 

Yes 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

36  man-months  (including  preliminary  investigation  phase) 

Development  Time:  End  Product 

System  not  fielded 

Documentation 

Yes,  extensive 

Configuration  Management 

Yes 

Conventional  SW  Interface 

Yes.  user  interface  and  support  environment 

Tools  Used-  Requirements  Analysis 

None 

Tools  Used:  Development 

In-house  developed  Expert  Sysvem  Shell 

Idols  Used:  Testing 

No 

Formal  'lest  Procedures 

Yes 

Testing  Criteria 

Experts  and  potential  users  ratings  of  system 

Test  Data 

'lest  scenario 

Self  Modifying  Code 

No 

II 


v 

yv<- 

&V 

.  ,N  ."V 


< 

-  ■ ,  • . " .  - .  vV 

>’>>$VSVVV 


u* i  *it  Vi  *it  *rf-ht  r 


4.1  Case  Study  Data 


Table  4.1-21  :  PAR  Government  Systems  Corporation  -  See  ami  Project  Enemy  Activity  (SPKA) 


FEATURE 


Problem  Category 
Domain 

Current  System  Phase 
Type  of  Funding 
Experienced  AI  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
( 'onvcutioual  SW  Interface 
Tools  Used:  Requirements  Analysis 
Tools  Used:  Development 
Tools  Used:  Testing 
formal  'Test  Procedures 
'Testing  Criteria 
'Test  Data 

Self  Modifying  Code 


APPLICATION 


Decision  aid 

Battle  situation  projections 
Completed  prototype 
Contract 
Yes 

3-6  computer  scientists  Sc  in-house  expert(s) 
Moderate 

Define  problem  and  test  system 
Yes 

Object  oriented 
Inheritance 
Yes 

Informal  -  project  staff 
N/A 


30  man-months 
System  not  fielded 
Yes,  extensive 
Yes 

Yes,  databases  and  support  environment 
None 
Flavors 


Experts  and  potential  user  ratings  of  system 
Scenario 


_ FEATURE _ 

Problem  Category 

Domain 

Current  System  Phase 
Type  of  funding 
Experienced  Al  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  Emd  product 
Documentation 
Configuration  Management 
Conventional  SW  Interface 
Tools  Used:  Requirements  analysis 
Tools  Used:  Development 
Tools  Used:  Testing 
Formal  Test  Procedures 
Testing  Criteria 


_ APPLICATION _ 

Test  equipment  assistant 

Countermeasure  systems 
Mature  prototype  -  funding  cut 
Internal  Research  ii  Development 
None 
3 

Low 

Knowledge  definition 
No 

Frames 

Frame-based  and  constraints 
No 

Informal 

Yes 

Yes,  user  interface  and  knowledge  base 
Yes 

6  months 
Not  applicable 

IR&D  plan,  listings,  frame  knowledge 
No 
No 
No 

Symbolics  environment  ii  some  internally  developed 

No 

No 

User  satisfaction 


Test  Data 


Sell  Mi>dd\  Itlg  ('mli' 


Daily  usage 
No 


■f 


4.1  Case  Study  Data 


Table  4.1-23  :  Schlumberger,  Inc.  -  Dipmeter  Advisor  System 


FEATURE 

APPLICATION 

Problem  Category 

Interpretation 

Domain 

Geology  -  dipmeter  logs 

Current  System  Phase 

Completed  -  commercial  system 

Type  of  Funding 

Commercial 

Experienced  AI  Staff 

Unknown 

Size  of  Development  Team 

Questionable  4-6  (assumption) 

Level  of  User  Involvement 

Active  in  system  definition 

Domain  Expert  Role 

Defined  Dipmeter  system  knowledge 

Formal  Development  Procedures 

No 

Type  of  Knowledge  Representation 

Rule-based 

Inference  Mechanism 

Forward  chaining 

Requirements  Analysis 

Yes 

Approval  Mechanisms 

Informal  (domain  expert) 

Iterative  Development  Process 

Yes 

Design  Changes 

Major  throughout  system  development 

Prototypes  Built 

Yes 

Development  Time:  Initial  Prototype 

18-22  months 

Development  Time:  End  Product 

4  years 

Documentation 

Informal  (Assumption) 

Configuration  Management 

Unknown 

Conventional  SW  Interface 

Yes 

Tools  Used:  Requirements  Analysis 

Yes 

Tools  Used:  Development 

Yes:  Strobe,  Impulse,  InterLISP 

Formal  Test  Procedures 

Yes 

Testing  Criteria 

User  acceptance;  integration  testing  with  the  system 

Test  Data 

Graphical  interface  with  scrolling  log  data 

Self  Modifying  Code 

Unknown 

4-25 
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FEATURE 


Problem  Category 
Domain 

Current  System  Phase 
Type  of  Funding 
Experienced  A I  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Reqii  i  rements  A  naly.sis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
Conventional  SW  Interface 
Tools  Used:  Requirements  Analysis 
Fools  Used:  Development 
Tools  Used:  Testing 
Formal  Test  Procedures 
'lesling  Criteria 
Test  Data 

Sell  Modifying  Code 


APPLICATION 


Decision  support 

Collaborative  decision  information  support 
Development 
Contractual 


Very  active 

Requirements  definition.  Analysis 
Yes,  internal 
Rule-based 
Production  system 
Yes 

Mostly  informal  except  requirements.  CM  approval  for  changes 


12  months 
Not  applicable 

Yes,  requirements  document,  design  document,  increment  plan 


Text  processing  programs 
DSDS  (Decision  Support  Development  System) 
None 

Alpha  and  beta  testing 

Experts  made  final  decision  on  system  performance 
Usage 


4.1  (  nsi*  Stmly  U.if.'i 


Table  4.1-25  :  Software  Architecture  and  Engineering,  Inc.  -  Sensitive  Financial  Analysis  Systen 


_ FEATURE _ 

Problem  Category 

Domain 

Current  System  Phase 
Type  of  Funding 
Experienced  A I  Staff 
Size  of  Development  Team 
Level  of  User  Involvement 
Domain  Expert  Role 
Formal  Development  Procedures 
Type  of  Knowledge  Representation 
Inference  Mechanism 
Requirements  Analysis 
Approval  Mechanisms 
Iterative  Development  Process 
Design  Changes 
Prototypes  Built 

Development  Time:  Initial  Prototype 
Development  Time:  End  Product 
Documentation 
Configuration  Management 
Conventional  SW  Interface 
Tools  Used:  Requirements  Analysis 
Tools  Used:  Development 
Tools  Used:  Testing 
Formal  Test  Procedures 
Testing  Criteria 
Test  Data 

Self  Modifying  Code 


_ APPLICATION _ 

Classification 

Financial  analysis 
Completed  -  production  system 
Contractual 


Active 

Involved  in  list  proress,  defined  schedule 
Yes,  internal 
Rule-based 
Production  system 
Yes,  internal  review 
No 

Yes,  incremental  system  development 
No 
Yes 

3  months 
8  months 

Not  required  by  customer.  Internal  notes. 

Informal 

Interface  with  conventional  software  via  Apollo  OS  commands 

None 

KES,  text  editor  and  KES  parser 
None 

Case  studies  with  known  outcomes 
Performance  judged  by  experts 
Test  case  studies,  individual  rules 
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4.1  Case  Study  Data 


Table  4.1-26  :  Texas  Instruments  Inc.  -  Production  Scheduler  Project 


FEATURE 

APPLICATION 

Problem  Category 

Scheduling  mechanism 

Domain 

Textile  liber  product  ion  and  inventory 

Current  System  Phase 

Project  terminated  prior  to  viable  prototype 

Type  of  Funding 

(  ommercial 

Experienced  AI  Staff 

Yes 

Size  of  Development  Team 

3 

bevel  of  User  Involvement 

Minimal 

Domain  Expert  Role 

Knowledge  and  constraint  definition. 

Formal  Development  Procedures 

No 

Type  of  Knowledge  Representation 

Frame- based 

Inference  Mechanism 

Algorithmic  mechanism  controlled  schedule  processing 

Requirements  Analysis 

No 

Approval  Mechanisms 

None 

Iterative  Development  Process 

Not  applicable 

Design  Changes 

Yes 

Prototypes  Built, 

No 

Development  Time:  Initial  Prototype 

Not  applicable 

Development  Time:  End  Product 

Not  applicable 

Documentation 

Informal  memos 

Configuration  Management 

Stamp  of  saved  Lisp  source  code  files 

Conventional  SW  Interface 

No 

Tools  Used:  Requirements  Analysis 

None 

Tools  Used:  Development 

Lisp,  windowing  forms  management  tool 

'Fools  Used:  Testing 

None 

Formal  Test  Procedures 

Planned  but  not  applicable 

'lesting  Criteria 

Interface,  schedule  checked  for  accuracy.  Both  tested  together 

Test  Data 

Schedule,  user  interface 

Self  Modifying  Code 

No 
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4.2  Evaluation  of  the  Case  Study  Data 

This  subsection  presents  an  analysis  of  the  case  study  data  depicting  generally  common  aspects, 
relational  trends  and  distinguishing  features. 

4.2.1  Common  Aspects 

The  twenty-six  case  study  tables  were  evaluated  as  a  single  group  in  an  attempt  to  identify  com¬ 
monalities  germane  to  the  development  process  regardless  of  features  such  as  current  project  phase, 
problem  category  or  domain.  In  general,  the  projects  appear  to  share  a  number  of  common  char¬ 
acteristics  as  delineated  below: 

•  Successful  systems; 

•  A I  experience; 

•  Small  development  teams; 

•  Participative  users; 

•  Knowledge  representation  schemes; 

•  Iterative  development  process; 

•  Rapid  prototyping; 

•  Use  of  development  tools;  and 

•  Static  code. 

All  participants  claimed  that  their  systems  were  successful  as  measured  by  some  criteria  such  as 
user  satisfaction,  expert  evaluation,  productivity  gains  and/or  demonstrating  the  feasibility  of  a 
new  technology.  In  addition,  the  project  teams  were  generally  comprised  of  5  or  less  people  and 
included  at  least  one  engineer  with  AI  experience.  The  users  generally  played  an  active  role  in 
system  development  and  their  acceptance  was  extremely  important  in  determining  whether  or  not 
the  system  was  a  success. 

The  development  process  for  all  projects  was  iterative,  including  design  changes  and  prototyping 
For  many  of  the  projects,  the  initial  prototype  was  completed  in  6  months  or  less.  The  knowledge 
representation  scheme  for  all  of  the  projects  was  based  on  rules,  frames  or  a  combination  of  both 

Most  of  the  twenty  projects  used  tools  during  the  development  phase.  Some  tools  were  built  by  the 
participants  but  many  of  the  tools  used  were  inherent  to  the  Al  workstation  Tools  were  generally 
not  used  in  support  of  requirements  analysis  or  testing. 

Lastly,  evaluation  of  the  case  studies  indicates  that  few  systems  include  self-modifying  code 
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4.2.2  Relational  Trends 

The  case  study  data  was  also  analyzed  in  separate  groupings  as  a  function  of  certain  relational 
characteristics  Those  groupings  that  appeared  to  indicate  significant  trends  are: 

•  team  size  versus  tools; 

•  lit  Ar  0  versus  commercially  funded  projects;  and 

•  user  involvement  as  a  function  of  project  phase. 

1'hese  relationships  and  apparent  features  are  discussed  in  the  following  subsections. 

4.2.2. 1  Team  Size  Versus  Tools 

The  literature  review  indicated  that  both  tool  power  and  development  team  size  were  areas  of 
difference  between  Al  and  conventional  software  development.  Analysis  of  responses  to  the  ques¬ 
tionnaire  and  further  review  of  reports  from  AI  development  teams  indicates  there  is  a  relationship 
between  the  two  characteristics.  As  noted  earlier,  the  trend  has  been  to  provide  tools  of  increased 
power  to  accomplish  specific  functions  after  knowledge  was  acquired  about  the  task.  Tools  have 
also  been  developed  to  shorten  system  development  time,  to  allow  small  teams  of  people  to  solve 
still  larger  problems,  and  to  automate  the  development  process  ensuring  some  higher  degree  of 
confidence  in  the  consistency,  quality,  or  reliability  of  the  programs. 

Some  very  specialized  tools  were  developed  or  purchased  to  perform  specific  system  functions 
such  as  data  base  interfaces,  or  micro-code  compilers.  Most  tools  were  developed  to  support  the 
development  team  efforts  and  were  aimed  at  a  better  program  implementation.  There  were  some 
tools  used  to  assist  in  the  testing  effort.  There  were  still  fewer  tools  used  in  the  fielded  or  mature 
production  phases  of  the  KBS  systems. 

In  general,  the  tools  supported  the  development  team  processes.  Tool  usage  was  not  as  prevalent 
in  the  other  stages  of  a  project.  Fewer  tools  were  purchased  than  were  developed.  More  tools 
were  developed  for  IR  A:  D  projects  than  for  contract  projects.  Tools  developed  in  the  academic 
community  were  adapted  for  use  in  IR  Sc  D  projects. 

4. 2. 2. 2  IF  &  D  Versus  Contract  Funding 

Of  the  twenty-six  case  studies,  twelve  were  funded  via  internal  research  ic  development  money, 
nine  were  commercially  contracted  and  five  contracted  by  the  DOD.  This  cross-section  of  the  data 
revealed  some  interesting  trends  within  the  three  groups. 

Among  the  IR  D  projects,  several  characteristics  were  observed.  Namely,  these  projects  more 
often  followed  structured  development  procedures  and  produced  more  written  documentation  than 
the  commercially  funded  projects.  In  addition,  initial  prototypes  were  generally  completed  earlier 
(within  fi  months)  when  compared  to  the  commercial  systems.  These  traits  seem  to  be  commensu¬ 
rate  with  obtaining  and  prolonging  management  support.  The  onus  is  on  the  project  staff  to  earn 
managerial  support  by  showing  enough  progress  to  satisfy  them  at  specified  time  intervals. 


4.2.3  Distinguishing  Features 

Among  the  commercial  projects,  expert  involvement  seemed  to  be  greater  when  compared  to  the 
IR  Sc  D  projects.  In  addition,  most  all  of  the  commercial  systems  were  developed  to  some  end  st  ale 
whereas  many  of  the  IR  it  D  projects  were  terminated  prior  to  completion.  It.  is  not  surprising 
that  in  the  commercial  sector,  there  is  a  greater  commitment  towards  developing  an  end  product 
and  not  as  much  competition  for  the  same  pool  of  funds  once  the  contract  has  been  awarded. 

The  projects  contracted  by  the  DOD  were  similiar  to  the  commercial  systems.  Along  with  displaying 
the  above  mentioned  characteristics,  they  also  produced  the  most  written  documents  of  the  three 
groups.  They  tended  to  use  some  form  of  standard,  such  as  the  phased  waterfall  development 
approach  or  a  formal  MIL-STD  specification  to  design  the  system.  If  not  adhered  to  strictly, 
formal  standards  were  at  least  used  as  a  guideline  for  system  documentation. 

4.2.2.3  User  Involvement  Versus  Project  Phase 

Observations  made  from  the  case  studies  indicate  that  user  and  domain  expert  involvement  are 
both  critical  to  the  success  of  KBS  software  development.  The  user’s  evaluation  of  a  system  de¬ 
termines  whether  continual  financial  commitment  can  be  obtained  throughout  the  development 
effort  and  helps  the  user  develop  realistic  expectations  of  ultimate  system  functionality  and  per¬ 
formance.  Users  tend  to  actively  participate  throughout  the  entire  system  development  effort . 
However,  according  to  the  study,  the  user’s  involvement  appears  to  increase  from  the  feasibility 
to  the  development  phase  and  continue  into  production  particularly  with  respect  to  testing  the 
system. 

The  domain  expert’s  knowledge  seems  to  be  mainly  required  during  feasibility  and  development  of 
the  prototype  in  order  to  define  the  requirements,  the  system’s  knowledge,  and  the  problem-solving 
techniques.  The  expert  also  helps  with  the  testing  effort. 

4.2.3  Distinguishing  Features 

In  contrast  to  common  aspects  and  relational  trends,  several  of  the  cases  reveal  features  that  are 
not  widely  reported  among  the  study  participants.  One  distinct  characteristic  has  to  do  with  the 
use  of  audio/visual  equipment  during  the  development  process.  Specifically,  during  the  knowledge 
acquisition  phase,  both  Northrop  Avionics  Division  and  Schlumberger-Doll  Research  tape  recorded 
interview  sessions  with  the  experts.  The  information  reported  was  then  captured  for  repetitive 


referencing  and  less  subject  to  recollection  augmented  by  the  knowledge  engineer’s  notes  alone  In 
addition,  the  inflections  used  by  the  experts  are  maintained  which  may  often  affect  the  knowledge 
engineer’s  interpretation  of  the  data. 

In  assessing  user  satisfaction  with  DSDS,  SA  Sc  E  video  taped  customer  reactions  while  using  the 
product.  This  approach  was  reported  to  be  more  conducive  than  asking  the  users  questions  about 
performance  while  they  are  simultaneously  acquainting  themselves  with  a  new  system. 

Lockheed-Ceorgia  Company’s  Pilot’s  Associate  is  being  developed  by  a  much  larger  team  than  the 
other  projects  in  the  survey.  The  distinguishing  characteristic  is  not  necessarily  the  size  of  the  team 
though.  A  more  prominent  feature  is  that  several  expert  systems  are  integrated  into  the  product 
Therefore  the  number  of  personnel  working  with  each  individual  expert  system  would  most  likely 
be  similar  to  the  size  of  the  development  teams  of  the  other  projects. 
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The  development  team  for  DEC’s  XCON  system  is  also  much  larger  than  the  others.  The  notably 
large  size  of  the  knowledge  base  (more  than  10,000  rules)  requires  much  support.  Several  knowl¬ 
edge  engineering  teams,  each  comprised  of  seven  to  eight  people,  perform  knowledge  acquisition 
and  representation  tasks,  continually  increasing  the  size  of  the  knowledge  base  as  new  products 
are  developed  and  existing  products  are  modified.  Groups  to  manage  administrative  duties,  user 
support,  and  technical  support  also  exist  to  provide  assistance. 

In  terms  of  managing  a  large  systems  project,  Northrop  Avionics  Division  strongly  recommends  a 
conventional  software  development  cycle  wherein  iterations  are  acceptable.  'Phis  technique  would 
involve  the  required  engineering  disciplines  and  implies  that  the  engineers  across  the  various  disci¬ 
plines  must  be  knowledgeable  in  AI  Because  a  military  systems  outlook  is  so  important,  knowledge 
engineers  who  are  not  experienced  with  large  avionics  systems  development  are  inadequate  for  the 
job  unless  the  problem  is  well-defined  and  well-bounded. 


4.3  SDI  Related  Issues/Implications 


The  twenty-six  case  studies  provide  an  experience  base  from  which  SDI  related  issues  and  im¬ 
plications  can  be  drawn.  The  most  notable  issues  are  acquisition  risk  reduction,  tools,  real-time 
processing  requirements,  project  management  and  testing. 

In  terms  of  reducing  the  risks  associated  with  acquiring  a  KBS  system,  it  is  prudent  to  examine  those 
aspects  that  are  common  to  all  case  studies.  Namely,  the  development  process  for  all  twenty-six 
projects  was  iterative  and  based  on  prototyping.  With  this  method,  the  requirements  specification 
and  the  design  are  expected  to  change  over  some  period  of  time  with  the  ultimate  goal  of  providing 
a  system  that  satisfies  the  users  needs.  In  addition,  it  was  noted  that  all  of  the  case  projects 
employed  a  common  set  of  knowledge  representation  schemes:  rules,  frames  (or  frame-like)  or  a 
combination  of  both  The  knowledge  representation  techniques  for  rules  and  frames  seem  stable 
and  would  not  present  risk  in  the  application  of  this  technology. 

In  the  case  studies,  reasoning  strategies  have  centered  on  search  and  inferencing  techniques.  There 
are  several  commercially  available  inference  engines  and  LISP  embedded  language  capabilities  (Sec¬ 
tion  3.4  5)  that  have  been  used  successfully.  Extensive  experience  with  the  tools  in  a  development 
environment  has  led  to  a  reduction  in  risk  associated  with  their  use  for  software  development.  Tool 
use  in  other  phases  of  a  KBS  project  has  not  been  extensive  and  represents  an  area  where  more 
effort  needs  to  be  directed. 

Since  all  the  case  study  participants  claimed  success  in  some  manner,  it  may  be  wise  to  scrutinize 
deviations  from  the  abeve  techniques  during  the  KBS  system  acquisition  process. 

An  important  issue  involves  the  nature  of  real-time  processing  that  is  likely  to  be  required  in  SDI 
applications  There  is  only  one  system  that  was  represented  as  having  real-time  requirements 
commensurate  with  SDI  needs,  and  it  was  never  tested  to  verify  that  it  met  those  requirements. 
The  requirements  on  KBS  software  in  the  SDI  system  are  likely  to  be  at  least  an  order  of  magnitude 
greater  than  for  a  system  where  the  operator  interaction  is  the  pacing  element  for  real-time. 

With  two  exceptions,  development  teams  have  been  uniformly  small  and  focused  on  a  single  project 
that  was  scoped  to  fit  their  capabilities.  It  is  expected  that  the  BM/C 1  AI  software  will  be  larger 
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and  more  complex  than  any  of  the  projects  reported  on.  There  are  questions  about  how  to  partition 
the  system  so  that  several  teams  can  work  on  the  problem,  and  ensure  that  the  fum  tional  interfaces 
will  work.  Project  management  is  also  an  issue  since  there  is  no  experience  available  in  terms  of 
managing  a  very  large  KBS  system.  Application  of  tools  to  support  management  decisions  in  the 
development  of  a  system  also  needs  further  study. 

The  testing  aspect  of  expert  system  development  is  not  well-defined  because  of  the  nondeterministic 
nature  of  the  solution  space.  Consequently,  those  case  systems  that  pertain  to  military  applications 
are  intelligent  assistants  or  decision  aids  as  opposed  to  autonomous  systems.  In  addition,  for  all 
of  the  cases,  the  test  data  is  based  on  either  hypothesized  inputs  or  actual  usage.  In  terms  of  SDI 
applications,  the  testing  aspect  has  several  implications  as  delineated  below. 

The  nature  of  a  decision  aid  is  that  the  responses  or  conclusions  reached  are  not  100%  accurate. 
Because  human  intervention  is  expected  in  reaching  a  final  decision,  some  degree  of  system  inac¬ 
curacy  is  acceptable.  For  SDI  applications,  however,  the  service  period  of  the  system  is  expected 
to  be  so  short  that  there  will  be  little  possibility  of  human  intervention  [49,  Parnasj. 

In  addition,  heuristic  programs  are  often  developed  by  trial  and  error  using  the  concept  of  build 
a  little,  test  a  little.V/hen  errors  are  encountered,  the  expert  is  asked  to  review  the  situation  and 
add  more  knowledge.  Because  heuristic  programs  may  exhibit  important  gaps  in  knowledge  at 
unexpected  times,  this  approach  would  not  be  acceptable  for  many  SDI  applications. 

Furthermore,  the  best  expert  system  simply  imitates  an  expert  exceptionally  well.  Since  humans 
are  not  perfect,  expert  systems  cannot  be  expected  to  reach  accurate  conclusions  all  of  the  time 

In  terms  of  test  data,  it  would  be  impossible  to  test  an  SDI  system  during  actual  use.  Furthermore, 
the  set  of  cases  that  could  be  hypothesized  would  not  come  close  to  covering  the  set  of  all  possible 
situations.  The  software  systems  addressing  SDI  objectives  will  be  significantly  larger  and  more 
complex  than  any  systems  built  to  date.  Consequently,  the  behavior  of  these  systems  under  all 
conceivable  circumstances  cannot  be  known  in  advance.  Another  major  factor  complicating  the 
prediction  of  potential  stimuli  is  the  use  of  enemy  countermeasures  of  which  there  is  no  current 
knowledge.  A  significant  aspect  that  has  not  been  addressed  is  how  to  test  a  system  for  robustness 
in  the  face  of  uncertainty. 

As  indicated  by  the  commentary  in  this  subsection,  continuing  research  is  required  in  terms  of  the 
application  of  KBS  to  the  SDI.  More  details  are  presented  in  Section  5.3. 
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5.1  Comparison  of  KBS  Development  Process  to  DOD-2167 

The  software  development  model  under  DOD-STD-2167  is  patterned  after  the  waterfall  model  pe- 
viously  discussed  and  includes  the  concepts  of  activities,  products,  reviews  and  baselines  to  further 
expand  the  waterfall  concept.  The  model  represents  software  development  from  the  government 
viewpoint,  and  as  such  visualizes  the  development  process  as  a  series  of  sequential  phases  with 
reviews  and  documentation  integral  to  each  particular  phase.  It  is  generally  recognized  by  both 
government  and  industry  that  software  development  is  not  a  well-bounded  sequential  process  but 
instead  consists  of  overlapping  phases. 

The  2167  waterfall  model  was  developed  to  provide  government  insight  into  software  development 
progress  and  to  provide  control  mechanisms  over  the  evolving  product  as  development  occurred. 
There  was  a  strong  configuration  management  influence  in  developing  the  model  which  resulted 
in  emphasis  being  placed  on  configuration  identification  through  the  concept  of  computer  software 
configuration  items(CSCI’s),  and  configuration  control.  Although  2167  provides  no  guidance  on 
what  constitutes  a  CSC  I,  a  companion  document,  M1L-STD-483,  does  provide  guidance  concerning 
CSCI  selection.  The  2167  requirements  for  baseline  establishment  and  control  provide  the  mecha¬ 
nism  for  controlling  requirements,  design,  and  code  as  it  evolves.  Associated  with  the  establishment 
of  baselines  are  various  reviews  aimed  at  assessing  software  development  progress  at  various  phase 
points  (e.g.,  requirements  analysis,  preliminary  design)  and  determining  readiness  for  baseline  con¬ 
trol.  The  documentation  produced  was  designed  to  be  a  natural  fall-out  of  the  activities  within  a 
particular  phase  of  development. 

This  ordered,  phased  approach  was  expected  to  improve  the  DOD  posture  for  developing  and 
supporting  software.  The  documented  2167  model  was  designed  to  provide:  visibility  and  control 
over  the  evolving  software  products;  a  quality  product  at  delivery;  and  a  proper  environment  for 
maintenance  and  modification  of  the  software  in  its  fielded  environment  Although  2167  is  t  new 
standard,  previous  developments  that  have  followed  the  principles  of  2167  have  generally  been 
successful  developments  and  have  provided  software  that  is  maintainable  and  supportable  when 
fielded. 

As  stated  previously,  the  KBS  system  development  process  is  a  highly  iterative  process  and  because 
of  this  most  models  represent  development  as  an  incremental  process  reflect  ing  btnhl  o  huh  .  It  * I 
a  Itllh  implementation.  Models  developed  for  expert  systems,  for  example,  clearly  define  the  A I 
system  development  process  as  a  continuous  one  |3.r>,r>6.  I  laves-  Noth,  Scow  n|  with  feedback  to  any  of 
the  previous  phases  of  development.  Contrast  this  with  the  phased  development  of  software  under 
I)OI)-STO-2I67  and  it  is  obvious  that  there  exists  incompatabilities  that  are  of  significance 

These  incompatabilities  are  not  significant  in  terms  of  activities  performed  during  development  but 
instead  are  mainly  related  to  the  use  of  prototyping  KBS’s  as  discussed  in  Section  3  Whereas  the 


1 


5.2  Interface  of  Conventional  and  KBS  Software 


2167  model  and  its  use  in  the  development  of  conventional  software  is  essentially  oriented  towards 
defining  the  total  requirements,  then  producing  a  design  followed  by  complete  implementation,  the 
KBS  approach  to  system  development  advocated  by  most  developers  follows  a  different  planned 
approach.  KBS  systems  typically  define  the  problem  domain,  then  begin  by  designing,  building 
and  testing  some  small  subset  of  this  domain.  Feedback  from  this  first  version  of  the  software  can 
then  be  incorporated  into  the  next  increment  with  its  added  capabilities  This  process  continues 
until  the  complete  domain  has  been  implemented  or  in  many  cases  never  ends  as  further  domain 
knowledge  is  collected. 

This  incremental  development  approach,  when  viewed  from  a  documentation,  review/andit,  and 
baseline  management  perspective,  leads  to  the  need  for  a  new  look  at  the  management  and  technical 
control  mechanisms  associated  with  KBS  system  development.  Questions  to  be  considered  are:  At 
what  point,  in  the  development  process  are  controls  established;  What  reviews  are  conducted  and 
when;  and  What  documentation  is  produced  and  when?  The  incremental  nature  of  the  KBS 
develo|  ment  process  must  be  factored  into  providing  answers  to  these  questions.  Further,  it  is 
expected  that  future  SDI  system  components  will  include  both  conventional  and  KBS  software. 
The  presence  of  both  types  of  software  will  require  that  the  development  approaches,  attendant 
reviews  •  audits,  configuration  controls,  and  documentation  be  compatable.  The  model  for  the  KBS 
development  process  is  a  part,  of  the  Volume  II  report. 


5.2  Interface  of  Conventional  and  KBS  Software 

Interface  of  KBS  wit  h  conventional  software  is  focused  on  two  areas  of  major  interest:  conventional 
programs  written  in  Ada  and  data  bases.  The  language  Ada  is  chosen  to  represent,  conventional 
software  because  of  its  standardization  and  projected  usage  rate  in  large  systems  of  interest.  Data 
bases  arc  chosen  as  another  focal  point  because  of  the  obviously  important  role  they  would  play  in 
any  battle  management  system  of  significant  size.  The  following  sections  discuss  the  management, 
functional,  testing  and  integration  aspects  of  the  interface  issues  between  KBS  and  conventional 
software. 


5.2.1  Management  Perspectives 

There  are  critical  issues  associated  with  the  sequencing  development  of  both  KBS  and  conventional 
software.  Management  planning,  reviews  and  direction  could  take  different  forms  depending  on 
whether  the  KBS  software  will  be  retrofitted,  integrated  during  development  or  planned  for  forward 
fit  with  a  conventional  software  system. 

Bridging  t  he  differences  between  the  different  and  possibly  competitive  technologies,  methodologies 
and  capabilities  requires  communications  and  contract  mechanisms  that  are  appropriate  to  the 
task  Implicit  in  the  communications  and  contracts  among  the  cooperating  groups  is  the  creation 
of  some  sort  of  meaningful  intermediate  products  that,  can  be  used  to  measure  performance,  risk, 
and  ini egi ation 

Cost  management  will  be  complicated  due  to  the  risk  factors  associated  with  both  the  development 
of  KBS  software  and  its  integration  with  conventional  software.  Costing  algorithms  that  have  been 
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applied  to  conventional  software  development  with  some  success  have  not  been  applicable  to  the 
KBS  development  efforts.  The  integration  of  both  software  types  in  a  single  system  may  also  subtly 
change  the  complexity  level  so  the  costing  algorithms  are  no  longer  valid  approximations  to  even 
the  conventional  efforts  cost. 

The  integration  of  KBS  and  conventional  software  in  a  system  will  require  the  extension  or  tailoring 
of  familiar  management  procedures  to  accomodate  the  characteristics  of  differing  development 
styles.  One  of  the  key  items  in  the  extended  procedures  will  be  the  identification  of  products  that 
can  be  used  to  track  the  expected  project  cost,  schedule,  performance,  and  accomplishment  goals 
versus  the  actual  progress. 


5.2.2  Implementation  Perspectives 

Issues  in  the  actual  implementation  of  KBS  and  conventional  software  include  specification,  design, 
interfaces  and  integration.  There  are  additional  decisions  to  be  made  in  the  systems  engineering 
allocation  process.  With  the  introduction  of  KBS,  there  has  to  be  an  assignment  of  intelligence  and 
knowledge  that  must  be  shared  by  the  user  of  the  system  and  the  computer  system.  System  level 
controls,  feedback  and  interface  will  have  to  undergo  more  cycles  of  iteration  than  is  common  for 
conventional  software  until  there  are  adequate  engineering  guidelines  and  knowledge  to  breakout 
these  functions  in  a  cookbook  manner. 

There  are  top  level  design  issues  including  synchronization  and  execution  of  programs  and  handling 
of  the  data  and  information  flows.  Using  present  technology,  the  questions  about  control  (low  apply 
predominantly  to  Ada  program  interfaces.  With  the  introduction  of  smart  memories  or  data  base 
machines  into  the  system  architecture,  con*~ol  flow  would  become  a  prominent  question  for  data 
base  interfaces  as  well.  Specific  questions  on  program  controls  are  phrased  as  follows.  How  do 
the  Ada  and  KBS  processes  know  when  to  start,  stop,  and  synch  or  at  least  align  in  a  common 
reference  frame?  The  questions  about  data  and  information  flow  apply  to  the  Ada  and  data  base 
interfaces  equally  and  include:  What  mechanism  is  required  to  handle  indeterminant  sized  amounts 
of  information  or  data?  What  do  the  data  interfaces  look  like  and  how  can  they  be  defined7 

Tlieie  ;ii  e  del  mled  let  Imn  al  quest ions  centering  around  language  eapalulit  ies  and  limit  at  tons  These 
questions  concern  computer  language  translation  and  expressivity,  integration,  task  allocation  am! 
efficiency.  Kxamples  of  issues  include:  Ada  execution  of  translated  KBS  programs,  the  ace,  n,  ia 
l  ion  of  self -modifying  programs;  control  passing  to  possibly  non-terininat  mg  program-  and  •  1 1  • 
tasking  for  concurrent  solution  searches. 

Integration  of  large  dissimilar  programs  historically  has  been  troublesome  I  In 
software  fault  tolerance  capability,  and  the  most  robust  of  comimicat  ions  and  ■ 
will  be  required  to  prevent  more  serious  difficulties.  Stability  of  interfaces  b. 
software  is  one  means  of  ensuring  better  communication  both  within  the  lechm.  . 
between  programs.  This  interface  definition  might  be  one  of  the  first  mt'im.  i 
under  management,  control. 
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5.2.3  Testing  and  QA  Perspectives 


5.2.3  Testing  and  QA  Perspectives 

The  integration  of  KBS  functionality  with  conventional  software  raises  many  additional  questions 
These  can  be  grouped  into  the  following  broad  classes.  How  should  nondclcrministic  processes  be 
tested?  In  expert  systems,  how  do  experts  become  certified?  With  ill  defined  requirements  how 
can  testing  procedures  be  derived? 

The  nature  of  the  application  may  itself  add  a  dimension  of  complexity  to  testing  and  QA  if  the 
proposed  operational  environment  has  many  unknowns  that  can  affect  the  nature  of  the  system  or 
its  responses.  Quality  Assurance  of  a  system  that  is  self  modifying  will  require  new  standards  and 
techniques. 


5.2.4  Comparison  of  Development  Techniques 

Contrasts  and  similarities  are  noted  between  the  development  approach,  and  the  intermediate  and 
final  products  from  KBS  and  Ada  developed  programs  that  have  significance  for  their  system  level 
integration. 

Ada  developed  programs  rely  on  support  from  the  language,  standards  for  development,  man¬ 
agement  and  review,  along  with  modern  software  engineering  practices.  KBS  programs  rely  on 
support  ,  tools,  and  power  from  the  languages  used,  highly  trained/skilled  developers  with  high 
standards  of  integrity  and  development  paradigms  that  have  no  analog  in  conventional  practices. 
There  are  different  problems  inherent  in  the  milieu  of  KBS  than  those  in  conventional  software  en¬ 
vironments.  These  include  dynamic  growth  of  the  data  structure,  control  of  the  operating  system, 
and  indeterminancy  of  results. 

Each  development  group  would  feel  more  comfortable  using  its  native  language  for  program  de¬ 
velopment  unless  the  application  was  more  naturally  expressed  in  the  other  language.  There  have 
only  been  preliminary  studies  in  the  area  of  productivity  and  release  error  rate.  The  most  recent 
available  study  used  a  controlled  experimental  method  to  evaluate  relative  productivity  on  the  four 
programming  tasks  of  pattern  matching,  maze  solution,  frame  editing,  and  a  heap  sort  [32,  Hattori]. 
The  experimental  results  indicate  doubled  productivity  for  LISP  over  Ada.  The  error  rate  reported 
in  this  study  was  slightly  lower  for  LISP.  To  minimize  language  related  problems,  management 
needs  to  enforce  the  necessary  standard  coding  conventions  and  a  rigorous  walkthrough  and  review 
process  to  eliminate  undocumented  tricks  or  bizarre  coding. 

There  are  several  similarities  which  are  recognizable  to  practitioners  in  the  different  groups.  KBS 
techniques  clearly  build  on  abstractions.  The  languages  used  in  KBS  efforts  are  more  powerful,  so 
smaller  units  of  code  are  required  for  most  classes  of  functions.  When  a  team  of  people  work  at  a 
single  terminal,  there  is  an  aspect  of  code  walkthrough.  Implementation  of  the  chief  programmer 
(rum  concept  is  necessary  to  avoid  team  members  stumbling  too  far  afield.  The  KBS  approach  to 
development  is  recognizably  bvild  a  little,  lest  a  little. 

Use  of  a  contract  basis  for  data  representation,  protocol,  budgeting  of  computer  system  resources 
and  retention  of  program  execution  control  within  the  conventional  software  package  are  methods 
that  could  be  used  bv  management  to  ensure  that  the  programs  can  be  successfully  integrated. 

Just  as  the  system  engineering  process  now  allocates  the  system  functionality  to  hardware  and  soft¬ 
ware.  there  will  have  to  be  a  similar  process  in  the  allocation  of  function  and  tasking  to  modules 
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5.2.5  Integration  with  Data  Bases 


with  or  without  intelligence  aa  well  as  some  specification  of  the  level  of  intelligence  that  is  re¬ 
quired.  Recognition  of  this  aspect  would  interface  the  KBS  and  conventional  software  development 
teams  with  the  system  engineering  team  to  produce  a  working  level  requirements  and  specification 
document. 

5.2.5  Integration  with  Data  Bases 

Integration  of  KBS  software  and  data  bases  is  critical  because  of  the  large  amounts  of  real  world 
data  that  are  accessed  by  BM/C  K  systems  of  the  size  contemplated  for  SDI.  The  two  critical  issues 
to  effective  integration  of  the  processes  are:  efficiency  in  data  flow  from  the  data  base  to  the  KBS 
application;  and  the  effective  direction  and  control  of  the  data  base  by  the  KBS  application.  It  is 
expected  that  the  strategies  to  effect  these  implementations  would  be  different  depending  on  the 
data  base  model,  the  domain  of  application  and  the  particular  type  of  KBS  software. 

5.2.6  Management  Implications 

From  a  management  viewpoint,  not  enough  is  known  about  the  KBS  development  process  to  have 
any  rules  of  thumb  for  gauging  how  well  development  is  proceeding.  The  identification  of  meaningful 
milestones  in  the  KBS  development  stream  are  difficult  to  assess  due  to  the  relative  inexperience 
in  management  control  of  developments  in  this  technology.  This  same  lack  of  experience  applies 
to  issues  across  the  board  such  as  the  contract  agreement  process,  standardization,  and  manpower 
planning. 

The  focus  of  the  statement  of  work  in  this  area  is  in  the  identification  of  a  standard  for  KBS 
software  interfacing  with  conventional  software,  and  in  the  management  implications  of  intragroup 
activities.  These  software  interfaces  include  the  management  and  distribution  of  data,  control  and 
intelligence. 

5.3  Application  of  AI  to  SDI  Issues 

The  application  of  AI  technology  to  SDI  J3M/C1  appears  to  be  concentrated  into  two  areas: 
applications  and  support  tools.  SDI  studies  conducted  to  date  have  primarily  concentrated  on  the 
support  tools  aspect  with  very  little  definition  in  the  applications  area. 

The  application  of  AI  to  support  tools  principally  affects  the  areas  of  planning  systems  and  knowl¬ 
edge  based  software  assistance  (KBSA).  The  KBSA  concept  of  providing  an  automated  assistant 
to  support  software  developers  is  envisioned  as  a  tool  that  will  contribute  significantly  to  software 
productivity  improvements. 

In  the  applications  area,  the  potential  for  applying  AI  is  only  limited  by  the  risk  envisioned  in 
developing  a  particular  application.  SDI  studies  to  date  have  left  the  specific  application  of  AI 
its  a  tn  hr  ih  lim  it  item.  However,  extensive  research  and  development  work  is  underway  in  the 
areas  of  decision  support  aids,  expert  maintenance  systems,  knowledge  based  signal  processing  and 
robotics.  All  of  these  research  efforts  have  the  potential  of  providing  significant  contributions  to 
SDI  /?*//<.". 
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Glossary 


Ada  -  ia  a  programming  language  that  was  designed  to  meet  the  needs  of  programmers  and  to 
embody  the  concept  of  design  methodologies  by  encouraging  and  supporting  good  design  and 
programming  practices. 

Artificial  Intelligence  •  The  science  of  making  machines  do  tasks  that  would  require  intelligence 
if  done  by  man.  An  approach  that  has  its  emphasis  on  symbolic  processes  for  representing  and 
manipulating  knowledge  in  a  problem  solving  mode. 

Automatic  Programming  -  The  ability  to  use  programs  to  automatically  generate  other  pro¬ 
grams. 

Backward-Chaining  •  An  inference  method  that  begins  with  a  goal  and  works  backwards  to  seek 
a  chain  of  premises  that  accounts  for  all  the  facts  at  hand. 

Causality  •  Inference  mechanism  based  on  the  understanding  of  the  structure  and/or  function  of 
a  given  device. 

Computer  Vision  -  Perception  by  a  computer,  based  on  visual  sensory  input,  in  which  a  sym¬ 
bolic  description  is  developed  of  a  scene  depicted  in  an  image.  Used  synonymously  with  image 
understanding  and  scene  analysis. 

Deduction  -  A  process  of  reasoning  in  which  the  conclusion  follows  from  the  premises  given. 

Demon  -  A  local  rule  or  procedure  which  is  triggered  upon  changes  to  specific  properties  in  a 
structured  knowledge  base. 

Domain  -  The  problem  area  or  region  of  knowledge(e.g.  bacterial  infections,  fault  diagnosis  or 
computer  configuration). 

DWIM  -  Do  What  I  Mean.  Part  of  the  InterLISP  environment  with  user  facilities  such  as  correcting 
mispelled  words,  variables  and  code. 

Empirical  Association  -  Inference  based  on  the  association  made  from  previous  experience  or 
observation. 

Event  Driven  -  A  forward  chaining,  problem  solving  approach  based  on  the  current  problem 
status. 

Expert  Systems  -  A!  systems  that  reflect  the  skill,  experience  and  judgement  of  humans  knowl¬ 
edgeable  in  a  particular  field. 

Explanation  Facilities  -  Ability  to  provide  a  trace  mapping  on  how  a  particular  problem  was 
solved. 

Exploratory  Programming  -  Conscious  intertwining  of  system  design  and  implementation. 

Failure  Tolerance  -  Term  used  in  testing  to  statistically  indicate  the  allowable  failure  rale.  For 
example,  to  meet  acceptance  criteria  for  a  given  system,  it  must  generate  accurate  results,  say,  95 
percent  of  the  time. 

Fault  Diagnosis  •  Determining  the  trouble  source  in  system. 


Glossary 


Forward  Chaining  -  An  inference  method  that  gathers  pieces  of  information  in  an  attempt  to 
build  forward  to  an  end  goal. 

Frame-Based  Knowledge  Representation  -  A  representation  method  that  clusters  closely  as¬ 
sociated  knowledge  about  a  class  of  objects  or  events.  A  frame  is  a  data  structure  that  associates 
one  or  more  features  with  an  object  in  terms  of  various  slots  and  particular  slot  values.  The  slots 
may  be  filled  by  values  or  perhaps  pointers  to  other  frames. 

Goal  Driven  -  A  problem  solving  approach  that  works  backwards  from  the  goal. 

Heuristics  -  General  rules  of  thumb  used  by  an  expert  to  process  information  in  a  particular 
problem  area. 

Inference  Engine  -  A  program’s  method  of  navigating  through  a  knowledge  base  in  an  attempt 
to  solve  a  problem. 

Inferential  Rule  -  An  associated  link  between  antecedent  conditions  and  resultant  beliefs  that 
permits  beliefs  to  be  inferred  from  valid  antecedent  conditions. 

Instantiation  -  Replacing  a  variable  by  an  instance  that  satisfies  the  system  or  the  statement  in 
which  the  variable  appears. 

Intelligence  -  The  degree  to  which  an  individual  can  successfully  respond  to  new  situations  and 
problems.  It  is  based  on  the  individual’s  knowledge  level  and  the  ability  to  appropriately  manipulate 
and  reformulate  that  knowledge  as  required  by  the  situation  or  problem. 

Intelligent  Assistant  -  An  A1  computer  program  (usually  a  knowledge-based  system)  that  aids 
a  person  in  the  performance  of  a  task. 

Knowledge  -  Knowledge  in  AI  is  basically  comprised  of  facts,  beliefs  and  heuristic  rules. 

Knowledge  Acquisition  -  The  process  of  extracting  and  formalizing  the  knowledge  of  an  expert 
for  use  by  an  expert  system. 

Knowledge  Base  -  Collection  of  facts,  inferences  and  procedures,  corresponding  to  the  types  of 
information  needed  for  problem  solutions. 

Knowledge  Based  Systems  -  Another  name  for  expert  system 

Knowledge  Engineer  -  A  software  engineer  specializing  in  the  techniques  of  knowledge  acquisition 
and  knowledge  representation. 

Knowledge  Representation  -  Formalized  structure  of  facts  and  heuristics  that  encompass  the 
descriptions,  relationships  and  procedures. 

Knowledge  Source  -  The  body  of  domain  knowledge  which  pertains  to  a  specific  problem,  (i.e., 
expert) 

LISP  -  A  principal  AI  programming  language.  By  definition,  LISP,  is  highly  recursive,  and  due 
to  an  untyped  and  applicative  framework,  highly  supports  symbolic  computing.  There  are  many 
LISP  dialects  which  include. 
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•  MacLISP;  and 

•  ZetaLISP. 

Metaknowledge  -  Knowledge  about  the  structure  of  and  how  to  use  knowledge. 

Metarule  -  A  rule  which  prescribes  the  manner  in  which  rules  should  be  used. 

Natural  Language  Understanding  •  The  response  by  a  computer  based  on  the  meaning  of  a 
natural  language  input. 

Object  Oriented  Programming  .  A  programming  approach  focused  on  objects  which  commu¬ 
nicate  by  message  passing.  An  object  is  considered  to  be  a  package  of  information  and  descriptions 
of  procedures  that  can  manipulate  that  information. 

Planning  Systems  -  An  expert  system  application  that  performs  planning  design  actions 

Predicate  Calculus  -  An  extension  of  propositional  calculus.  In  predicate  calculus,  elementary 
units  are  called  objects.  Predicates  are  statements  about  objects. 

Production  Rule  -  A  modular  knowledge  structure  representing  a  single  chunk  of  knowledge, 
usually  in  an  If-Then  or  Antecedent-Consequent  form. 

PROLOG  -  An  AI  or  symbolic  programming  language  that  is  based  on  predicate  calculus 

Prototype  -  An  initial  model  or  system  that  is  used  as  a  basis  for  building  future  systems 

Rapid  Prototyping  -  An  approach  used  in  system  development  to  quickly  generate  a  working 
model  that  simulates  a  response  to  a  simplified  version  of  the  problem  at  hand.  Through  incremental 
development,  the  simplified  model  is  made  increasing  complex  in  ultimate  response  to  the  problem 
statement. 

Robotics  -  One  AI  research  branch  that  is  involved  with  allowing  computers  to  see  and  manipulate 
objects  in  a  specific  environment. 

Rule-Based  Knowledge  Representation  -  A  representation  method  comprised  of  a  set  of  pro¬ 
duction  rules  based  on  a  situation  and  an  action. ..e.g.  If  Antecedent  -  then  Consequent  type  of 
structure.  A  rule  is  a  conditional  statement  consisting  of  a  part  comprised  of  one  or  more  if  A auses 
which  establishes  conditions  that  must  apply  if  a  second  part,  composed  of  one  or  more  then  clauses 
is  to  be  acted  upon. 

11  Iilem't  -  A  set.  of  rules  that  constitutes  a  module  of  heuristic  knowledge. 

Satisfice  -  Achieve  a  solution  that  satisfies  all  imposing  constraints. 

Semantic  -  Specifies  the  meaning  or  significance  of  a  symbolic  expression. 

Semantic  Net  -  A  knowledge  representation  scheme  composed  of  nodes  describing  objects,  and 
links  describing  relationships  between  nodes. 

Shell  -  Software  programs  used  to  develop  expert  systems. 

Slot  -  A  description  of  an  object  in  a  frame.  Slots  may  correspond  to  features  such  as  name, 
definition  or  creator,  or  may  represent  a  value. 

Syntactic  -  Specifies  the  form  or  structure  of  symbol  expressions. 
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Tools  for  Knowledge  Engineering  -  Programming  systems  that  facilitate  t  he  process  of  building 
expert  systems.  Particular  emphasis  is  placed  on  generic  task  packages  and  very  high-level  languages 
for  heuristic  programming. 

Truth  Maintenance  -  The  task  of  preserving  consistent  beliefs  iti  a  reasoning  system  whose  beliefs 
change  over  time. 
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Acronyms 


ABI/lnform 

APR 

AI 

ART 

ATO 

ATP 

ATR 

BM/C' 

CBD 

CBTAO 

Cl 

CM 

CMS 

CMU 

COMPASS 

CPCI 

CPL 

CPU 

CSCI 

DARPA 

DART 

DATA 

DBMS 

DEC 

DID 

DLL 

DOD 

DP 

DPMA 

DROLS 

DSDS 

DTIC 

KCM 

K( 

El  A 

ESP 

KSTAS 

ETI 

PDR.S 

PUL 

IIOL 

IBM 

IBM  FSC 


American  Business  Information 
Air  Force  Regulation 
Artificial  Intelligence 
Automated  Reasoning  Tool 
Air  Tactical  Operations 
Acceptance  Test  Plan 
Automatic  Target  Recognition 

Battle  Management/Command  Control  and  Communications 

Commerce  Business  Daily 

Cost  Benefit  of  Tactical  Air  Operations 

Configuration  Item 

Configuration  Management 

Configuration  Management  System 

Carnegie-Mellon  University 

Central  Office  Maintenance  Printout  Analysis  and  Suggest  System 

Computer  Program  Configuration  Item 

Constraint  Propagation  Language 

Central  Processing  Unit 

Computer  Software  Configuration  Item 

Defense  Advanced  Research  Projects  Agency 

Duplex  Army  Radio/Radar  Targeting 

Decision  Aids  for  Target  Aggregation 

Data  Base  Management  System 

Digital  Equipment  Corporation 

Data  Item  Description 

Data  Layout  Language 

Department  of  Defense 

Data  Processing 

Data  Processing  Management  Association 
Defense  RDT&E  On-Line  System 
Decision  Support  Development  System 
Defense  Technical  Inlormation  Center 
Electronic  Countermeasures 
Embedded  Computer  Systems 
Electronic  Industries  Association 
Expert  Software  Pricer 

Expert  System  for  Target  Attack  Sequencing 
Expert  Technologies,  Inc 
Fault  Diagnosis  and  Resolution  System 
Frame  Representation  Language 
High  Order  Language 

International  Business  Machines  Corporation 
International  Business  Machines  Federal  Systems  Group 
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1NSPEC  Information  Services  in  Physics,  Electrotechnology  Computers  ami  Control 

1R&D  Internal  Research  and  Development 

1RAD  Internal  Research  and  Development 

ITACC  Integrated  Tactical  Air  Control  Center 

lVAcY  Independent  Verification  and  Validation 

JLC  Joint  Logistics  Commanders 

KAR  Knowledge  Acquisition  Rules 

KBS  Knowledge  Based  System 

KBSA  Knowledge  Based  Software  Acquisition 

KEE  Knowledge  Engineering  Environment 

KES  Knowledge  Engineering  System 

LAS  Lockheed  Aircraft  Service  Company 

LES  Lockheed  Expert  System 

LGC  Lockheed-Georgia  Company 

MIT  Massachusetts  Institute  of  Technology 

NASA  National  Aeronautics  and  Space  Administration 

NSF  National  Science  Foundation 

NTIS  National  Technical  Information  Service 

OS  Operating  System 

PDR  Preliminary  Design  Review 

PDS  Problem  Definition  Statement 

PGSG  PAR  Government  Systems  Corporation 

QA  Quality  Assurance 

RADC  Rome  Air  Development  Center 

R&D  Research  Ac  Development 

SAAcE  Software  Architecture  Ac  Engineering,  Inc. 

SDI  Strategic  Defense  Initiative 

SFAS  Sensitive  Financial  Analysis  System 

SPEA  See  and  Project  Enemy  Activity 

SQR  System  Quality  Review 

SSE  Software  Systems  Engineering  Directorate 

STAMP  System  Testablility  and  Maintenance  Program 

TI  Texas  Instruments,  Inc. 

TMS  Truth  Maintenance  System 

UICP  Unidentified  Command  Post 
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Appendix  A 

Software  Development  Problems 


This  appendix  isolates  software  related  problems  that  surface  in  both  conventional  software  devel¬ 
opment  and  in  artificial  intelligence  systems  development.  The  following  software  categorization 
was  extracted  from  a  report  issued  by  the  Department  of  Defense  (DOD)  Joint  Services  Task  Force 
on  Software  Problems,  Report  of  the  DOD  Task  Force  on  Software  Pivblems.  One  purpose  of  that 
study  was  to  identify  conventional  software  problems  associated  with  embedded  computer  software. 

1.  Life  Cycle: 

(a)  Requirements; 

(b)  Management; 

(c)  Acquisition; 

(d)  Product  Assurance;  and 

(e)  Transition. 

2.  Environment: 

(a)  Disciplined  Methods; 

(b)  Labor  Intensive; 

(c)  Tools; 

(d)  Reinvention;  and 

(e)  Capital  Investment. 

3.  Software  Product: 

(a)  Doesn’t  Meet  the  Need; 

(b)  Software  Metrics; 

(c)  Design  Attributes; 

(d)  Documentation;  and 

(e)  Immutable  Software. 

4.  People: 

(a)  Skills; 

(b)  Availability;  and 

(c)  Incentive. 
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Appendix  A  Software  Development  Problems 


A  short  definition  of  each  category  and  subcategory  is  provided  followed  by  tables  of  conventional 
and  A1  systems  problems  that  were  identified  in  the  following  bibliographic  sources: 

1.  Sources 

(a)  Acquisition  and  Support  of  Embedded  Computer  System  Software,  September,  1981. 

(b)  DOD  Weapon  Systems  Software  Management  Study,  June,  1975. 

(c)  DOD  Weapon  Systems  Software  Acquisition  and  Management  Study,  Volumes  1  and  II, 
May  -  June,  1975. 

(  I)  Final  Report  of  the  Joint  Logistics  Commanders’  Workshop  on  Post.  Deployment  Software 
Support  (PDSS)  for  Mission  Critical  Computer  Software,  Volume  II,  June,  1975. 

(e)  Proceedings  of  the  Joint  Logistics  Commanders  Joint  Policy  Coordinating  Group  on 
Computer  Resource  Management,  November  1,  1981. 

(f)  Report  of  the  DOD  Task  Force  on  Software  Problems,  Third  Draft,  July  15,  1982. 

(g)  Report  of  the  USAF  Scientific  Advisory  Board,  December,  1983. 

(h)  Suggestions  for  DOD  Management  of  Embedded  Computer  Software  in  an  Environment 
of  Rapidly  Moving  Technology,  March,  1982. 

(i)  A  Practical  Guide  to  Designing  Expert  Systems,  Shalom  M.  Weiss  and  Casimir  A.  Ku- 
likowski. 

(j)  Artificial  Intelligence,  Patrick  Henry  Winston. 

(k)  Building  Expert  Systems,  Frederick  Hayes-Roth,  Donald  A.  Waterman,  and  Douglas  B. 
Lenat. 

(l)  “Computers  and  Information  Technology  in  the  Year  2000  -  A  Projection”,  Stephen  F. 
Lundstrom  and  Ronald  Lilarsen,  IEEE,  September  1985. 

(m)  Expert  Systems,  Paul  Harmon  and  David  King. 

(n)  “Expert  Systems:  Where  Are  We  And  Where  Do  We  Go  From  Here?”,  Randall  Davis, 
MIT  AI  Laboratory,  June  1982. 

(<>)  IEEE  1084  Workshop  on  Principles  of  Knowledge  Based  Systems,  IEEE  Computer  Soci¬ 
ety. 

(p)  “On  the  Development  of  Commercial  Expert  Systems” ,  Reid  G.  Smith,  The  AI  Magazine, 
Fall  1984. 

(q)  Rule-Based  Expert  Systems,  Bruce  G.  Buchanan  and  Edward  H.  Sbortliffe. 

(r)  “Starting  a  Knowledge  Engineering  Project:  A  Step  by  Step  Approach”,  Mike  Freiling, 
Jim  Alexander,  Steve  Messick,  Steve  Rehfuss,  and  Sherri  Shulman. 

(s)  The  A I  Business:  The  Commercial  Uses  of  Artificial  Intelligence,  Patrick  H.  Winston 
and  Karen  A.  Prendergast. 

(i )  Tin  Artificial  Intelligence  Expcru  nee,  Susan  J.  Scown. 

(n )  The  Handbook  of  Artificial  Intelligence,  Volumes  I-II,  Avron  Barr  and  Edward  A.  Feigen- 
baum. 
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A.l  Life  Cycle 


A.l  Life  Cycle 


Life  cycle  refers  to  the  relationship  among  activities  of  computer  software  development  from  its 
inception  to  retirement.  The  section  is  divided  into  five  major  categories:  Requirements,  Manage¬ 
ment,  Acquisition,  Product  Assurance  and  Transition. 


A. 1.1  Requirements 

The  requirements  phase  of  the  software  life  cycle  consists  of  the  analysis  and  definition  of  a  system 
as  defined  by  the  user.  As  such,  communication  between  users,  support  agents  and  acquisition 
agents  is  essential  to  the  success  of  the  final  system.  Oftentimes,  the  requirement  document  does 
not  accurately  define  the  product;  or  perhaps  uses  ambiguous  wording.  Inadequate  requirement 
information  results  in  costly  schedule  changes  to  hardware  and  software.  For  itemized  problems 
see  tables  A.l.6-1  and  A.l.6-2. 


A. 1.2  Management 

The  management  of  software  life  cycle  includes  activities  such  as  planning,  monitoring  of  schedules 
and  budgets,  assigning  responsibilities,  tracking  software  projects  and  effective  plans  for  software 
activities.  For  itemized  problems  see  tables  A.  1.6-3  to  A. 1.6-5. 


A. 1.3  Acquisition 

Software  acquisition  is  difficult  to  specify  because  the  rules  for  acquiring  software  have  historically 
been  based  on  the  rules  for  obtaining  hardware.  Some  major  aspects  of  acquisition  are: 

•  Management  of  System  Interfaces  -  Good  interfaces  must  be  defined  to  guarantee  a  Miiooth 
integration; 

•  Project  Status  Reports  -  Documentation  for  project  status; 

•  Documentation  Requirements  -  Documentation  where  applicable; 

•  Software  Change  Control  -  Software  must  be  carefully  managed  especially  where  changes 
occur; 

•  Reliance  on  Contractors  -  Several  contractors  are  frequently  responsible  for  managing  the 
software  process;  and 

•  Design  for  the  Complete  Life  Cycle  -  All  resources  needed:  hardware,  software,  documen  ation, 
tools,  personnel,  etc.  for  the  life  cycle  should  be  clearly  identified 

For  itemized  problems  see  tables  A.  1.6  6  and  A.  1.6  7. 
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A.  1.4  Product  Assurance 


\  I  I  Product  Assurance 
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(Mill  li\el  t'l  dll'  It'll',. Ilr  lilt'  i\ilr  lllglli‘1  t|llrtlll\  HollMilli-  |H  Hilllrittl  lllr  r.tlllrt  <llilr|'rltilrlll 

vrrilii  ati.'ii  hi. I  \:,lii|iil  it'll  "I  :i  system  ""  ttrs  in  llir  lilo  i\tle  I  *  >  |'ii'»i'l"  n  t|unli(\  |>i"'lll'(, 
adequate  resources  for  lost  planning  along  with  organir.ril  i  oiiiiiitinii  nl  ion  bclnccn  llir  nt  qiiwil  ion 
agent,  developmenl  contractor,  lost  agon! .  support  agent  and  user  an-  rarwnlinl  For  itrinirnl 
problems  see  lal'lrs  A.  1.6  Sand  A. I  .6  ‘.I. 

A. 1.5  Transition 

Transit  ion  refers  to  the  change  in  the  software  life  cycle  from  one  activity  to  another.  Two  examples 
involve  the  more  difficult  transitions  from  exploratory  research  to  engineering  development  and 
from  development  to  support  (also  known  as  maintenance).  Problems  occur  because  the  transition 
from  exploratory  research  to  engineering  development  is  characterized  by  new  and  rapidly  changing 
technology,  and  contrarily,  maintenance  requires  a  stable  technology  base.  For  itemized  problems 
see  tables  A.  1.6  10  and  A.  1.6- 11. 

A.  1.0  Life*  Cycle  Problem  Tables 

The  following  tables  show  the  problems  which  surfaced  in  each  of  the  Life  Cycle  categories. 
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A. 1.6 


Life  Cycle  Problem  Tables 
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Table  A.  1.6-1:  Conventional  Software  Requirements 
PROBLEM  STATEMENT  SOURCE(S) 

Poor  Communication  Skills.  f 

Diverse  interpretation  of  requirements.  c  d  f 

Requirements  defined  according  to  cost  and  schedule  con-  be  f 

straints. 

Inadequacy  of  English  prose  for  software  requirements.  c  f 

Excessive  detail  in  requirements  allocation.  b  f 

System  design  characteristics  appear  in  requirements  docu-  f 

mentation. 

Software  is  severely  impacted  by  requirement  changes.  a  b  c  d  e  f 

Poor  cost  estimation  analysis.  c  d  e  f 

Hardware  changes  made  with  little  knowledge  of  software  im-  b  f 

plications. 

Requirement  changes  are  costly.  abed  f 

Lack  of  valid  engineering  data  to  define  what  is  cost  effective  f 

for  support  requirements. 

A  support  agent  is  not  selected  until  late  in  process.  b  f 

No  operational  concept  developed.  b  f 

Concurrency  of  the  configuration  of  the  trainer  system  does  f 

not  always  correspond  to  that  of  the  primary  weapon  sytem. 

Protection  of  classified  software.  f 

Security  causes  increased  complexity  due  to  the  number  of  f 

systems  and  levels  of  security  required. 

Non-standard  systems  must  be  able  to  interface  with  others.  f 

Requirements  are  not  concrete  before  development  begins.  b  c 

Management  budgets  and  schedules  the  software  project 
without  a  true  understanding  of  the  requirements. 

There  exists  an  inadequate  understanding  of  the  product  to  b  c 
be  developed. 

Systems  continue  to  grow  in  complexity.  a  b  d  e 
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A.  1.6  lJfe  Cycle  Problem  Tables 


Table  A.  1.6 -‘2:  Artificial  Intelligence  Requirements 


PROBLEM  STATEMENT 


SOl)RCE(S) 


1.  Requirements  oriented  methods  and  intu¬ 
itions  learned  in  the  development  of  other 
types  of  software  do  not  carry  over  well  to 
the  knowledge  engineering  task. 

2.  A  major  difficulty  in  the  application  of  Arti-  1 

ficial  Intelligence  (Al)  systems  is  in  obtaining 

a  complete  specification. 

3.  Problem  Statements  are  inconsistent  or  in-  k  m 

complete  in  matching  requirements  to  the  fi¬ 
nal  product. 

4.  Poor  communication  between  engineers  and 
experts. 

5.  Knowledge  principles  and  problem-solving  I  m 

techniques  are  difficult  to  translate  into  a 

knowledge  base  to  be  used  by  an  expert  sys¬ 
tem 

6.  The  knowledge  is  often  ill-specified  because  i 
the  expert  cannot  always  express  exactly  what 

he  knows  about  his  domain. 

7.  When  available  at  all,  experts  have  little  time  i 
at  their  disposal. 

8.  The  knowledge  acquisition  phase  is  one  of  the  i 
most  difficult  and  time  consuming  phases  of 
expert  system  building. 

9  Problem  in  determining  the  kind  of  knowledge  i  j  k 
required  for  an  AI  system. 

Overlapping  AI  References:  2(Management),  7(Design  Attributes). 
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A.  1.6  Life  Cycle  Problem  Tables 


Table  A.  1.6-3:  Conventional  Software  Management 


[  PROBLEM  STATEMENT 

SOURCE(S) 

1. 

Lack  of  skilled  software  program  managers. 

b 

f 

g  h 

2. 

Air  Force  uses  inexperienced,  junior  officers  to  direct  pro¬ 
grams. 

f 

3. 

Misunderstanding  of  acquisition  concepts. 

b 

f 

4. 

Belief  that  hardware  and  software  can  be  developed  and  sup¬ 
ported  by  different  concerns  without  strong  engineering  in¬ 
fluence. 

f 

5. 

Prototyping  used  to  sell  a  system  but  does  not  achieve  its 
goal  of  risk  reduction. 

f 

6. 

Incompatibilities  between  State  Department  and  DOD  deci- 

c 

r 

sions. 

7. 

Lack  of  communication. 

b 

c. 

r 

8. 

Problem  with  the  transfer  of  technological  issues  and  the  clas¬ 
sification  of  data. 

c 

r 

9. 

No  clear  assignment  of  responsibility  and  authority  given  to 
those  working  for  management. 

r 

10. 

Lack  of  useful  data  on  projects  making  progress  difficult  to 

c 

r 

measure. 

11. 

Lack  of  historical  metrics  and  models. 

b 

c 

d  r 

12. 

Vague  requirements  definitions  for  complex  systems. 

c 

r 

13. 

Lack  of  tracking  tools  and  planning. 

c 

r 

14. 

Need  for  thorough  software  Development  Plan  to  check  va¬ 
lidity. 

c 

f 

15. 

Lack  of  performance  monitors  and  modeling  tools. 

c 

f 

g 

16. 

Management  budgets  and  schedules  without  a  true  under¬ 
standing  of  the  requirements. 

g 

17. 

Often  unrealistic  budgets  are  drawn  up  just  to  win  a  contract 
proposal. 

a  b 

g 

18. 

Inaccuracies  in  software  cost  estimation  models  and  tech- 

a  b 

c 

<1  e 

g 

niqucs. 

19. 

Contractual  budget  commitment  is  often  made  before  Pre¬ 
liminary  Design  Review(PDR)  occurs. 

g 

20. 

Software  is  not  reported  properly  in  large  system  status  re¬ 
ports. 

b 

c 

g 

21. 

High  risk  software  is  not  addressed  in  reports  and/or  reviews. 

b 

c 

g 

t]  I, 


Jl. 


PROBLEM  STATEMENT 


Software  status  reports  are  too  detailed  to  provide  useful  in¬ 
formation  or  too  costly  to  be  prepared  correctly. 

Insufficient  time  for  high  level  management  to  respond  to 
potential  problems  indicated  in  status  reports. 

Good  advice  is  sometimes  ignored  by  program  offices. 

Cost  information  is  rarely  correlated  with  technical  informa¬ 
tion  for  management  purposes. 

Management  needs  more  information  provided  to  identify 
where  DOD  software  costs  are  occurring. 

A  high  turnover  rate  of  military  software  management  per¬ 
sonnel  occurs  in  many  program  offices. 

Very  little  knowledge  available  to  acquisition  managers  for 
selecting  the  hardware,  software,  and  firmware  configuration 
items  of  a  system. 
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Table  A.  1.6-5:  Artificial  Intelligence  Management 

1.  Existing  AI  systems  cover  too  narrow  a  range  mo  q  s  t  u 

of  expertise. 

2.  Lack  of  sufficient  funding  of  AI  projects  exists.  r  t 

3.  Lack  of  strong  management  to  guide  an  AI  l 

project  through  completion  causes  difficulties. 

4.  Do  not  know  how  to  manage  the  transfer  of  p 

progressive  and  evolutionary  technology. 

5.  Knowledge  engineering  is  often  considered  by  r 

management  to  be  a  technology  that  is  far  too 

difficult  to  even  attempt. 

6.  There  are  difficulties  in  scaling  from  the  cur*  I 

rent  project  sizes  to  large  knowledge  based 
systems. 

Overlapping  AI  References:  l(Requirements),  3(Requirements),  1  (Acquisition),  1  (Product  As¬ 
surance),  l(Disciplined  Methods),  l(Metrics),  2(Metrics),  1( Design  Attributes),  7(Design  At¬ 
tributes),  l(Skills),  1  (Availability). 
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A. 1.6  Life  Cycle  Problem  Tables 


Table  A.  1.6-6:  Conventional  Software  Acquisition 


PROBLEM  STATEMENT 


SOURCE 


Many  software  contracts  are  inappropriate  as  they  are  de¬ 
signed  for  hardware. 

Changing  requirements  are  not  controlled. 

Software  development  tools  are  not  required  deliverables. 

No  funds  for  software  tools  or  documentation. 

Software  cannot  be  carefully  tracked  or  measured. 
Documentation  requirements  are  often  starved  for  funds. 

Documentation  is  over-whelming  in  volume  but  marginal  in 
value. 

Requirements  and  design  changes  are  not  managed  well,  if  at 


Design  changes  are  rarely  made  with  the  complete  life  cycle 
in  mind. 

Decisions  made  with  little  consideration  of  cost  over  the  life 

cycle. 

Even  when  software  is  not  a  primary  cost  or  deliverable,  it 
can  have  a  large  impact  on  system  costs  and  schedules. 

Need  to  improve  software  acquisition  management  with  more 
detail  in  early  stages  of  development. 
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A.  1.6  Life  Cycle  Problem  Tables 


Table  A. 1.6-7:  Artificial  Intelligence  Acquisition 


SOlUtCK(S) 


PROBLEM  STATEMENT 


1.  The  rapid  prototyping  method  does  not  offer 
a  clear  guidance  on  how  to  produce  a  well- 
engineered  commercial  product. 

2.  With  rapid  prototyping,  the  system  is  always 
in  a  state  of  flux. 

3.  A  precise  set  of  tools  needed  to  solve  a  prob¬ 
lem  cannot  be  identified  before  the  implemen¬ 
tation  phase. 

4.  Representations  at  too  specific  or  at  too  low 
a  level  yield  problems  in  integration,  mainte¬ 
nance  and  extensibility. 


Overlapping  AI  References:  3(Requirements),  4(Management),  l(Disciplined  Methods). 
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PROBLEM  STATEMENT 


Sn|i|U‘K(S) 


1  Lack  of  criteria  to  determine  how  much  testing  is  necessary  a  f 

at  each  stage  of  software  development. 

2.  Testing  is  often  governed  by  the  time  and  money  available.  a  f 

3.  Test  tools  are  not  a  real  measure  of  a  products  assurance.  c  f 

4  The  testing  of  some  requirements  is  extremely  difficult  to  f 

simulate 

5.  Software  test  planning  does  not  receive  an  adequate  share  of  a  f 

total  resources. 

6.  Incomplete  or  vague  functional  specifications  cause  errors  dis-  a  b  f 

covered  only  during  testing. 

7.  Software  testing  is  not  always  provided  throughout  the  entire  a  c  f 

program  life  cycle. 

8.  Independent  Verification  and  Validation  (IVAtV)  strategies  f 

are  costly. 

9.  Errors  exist  in  software  after  deployment  to  the  user  com-  d  h 

mand  regardless  of  the  testing  effect. 

10  There  exists  a  high  potential  for  a  significant  lapse  in  support  b  h 

during  early  deployment  stages. 

1 1  Testing  effort  requires  a  significant  amount  of  money.  b 


A.  1.6  Life  Cycle  Problem  Tables 


Table  A.  1.6-9:  Artificial  Intelligence  Product  Assurance 
PROBLEM  STATEMENT  |  SOUliCE(S) 


1.  There  exists  a  lack  of  knowledge  regarding  o  q  t  u 

testing  of  AI  systems. 

2.  AI  systems  have  no  independent  means  of  k 

checking  whether  their  conclusions  are  reason¬ 
able. 

3.  Experts  should  be  evaluated  as  well  as  the  sys-  q 

terns. 

4.  Acquisition  organization  and  structure  of  I  f 

knowledge  is  a  manual  process  with  the  re¬ 
quirement  or  detailed  quality  assurance. 

Overlapping  AI  References:  5(Requirements),  6( Requirements),  6(Design  Attributes),  7(Design 

Attributes),  8(Design  Attributes). 
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Table  A.  1.6-10:  Conventional  Software  Transition 
PROBLEM  STATEMENT  _ 

Rapid  changes  in  technology.  a  l> 

Post  engineering  data  becomes  obsolete  very  quickly. 
Microprocessors  and  firmware  are  forced  under  software  or 
hardware  guidelines. 

Need  to  develop  a  unique  policy  for  microprocessors  and 
firmware. 

Need  for  flexibility  to  adapt  to  technological  change. 

Inability  to  develop  and  validate  system  and  tactical  software 
requirements  and  evaluate  doctrinal  problems. 

Insufficient  acceptance  testing  performance  on  a  regular  basis 
for  automated  systems. 

Necessary  to  develop  and  support  automation. 

Need  for  a  single,  integrated  set  of  procedures,  guidelines, 
and  standards  for  development  of  automated  systems. 

Lack  of  identification  and  use  of  an  effective  procedure  for  b 
system  development. 

Automated  Systems  have  poorly  defined  function  and  inter¬ 
face  requirements  specifications. 

Automated  Systems  lack  proper  modularization,  use 
machine-orientated  languages,  and  are  inadequately  docu¬ 
mented. 

Most  Research  Si  Development  (R&D)  work  is  not  aimed  at 
a  specific  weapon  system. 

Rarely  is  the  transition  made  from  the  useful  techniques  and 
tools  of  R&D  to  operational  systems. 

Post  Deployment  Software  Support  has  to  reorient  its  work  a  b 
force  from  traditional  logistics  and  maintenance  functions  to 
advanced  technology  support. 

Lack  of  discipline  in  software  development  methodology.  b 

Delays  in  delivery  of  software  has  a  large  impact  on  system 
costs  and  schedules. 

Need  to  educate  involved  organizations  before  useful  output 
is  obtained. 

Need  a  specification  or  set  of  criteria  to  aid  in  the  documen¬ 
tation  selection  process. 
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PROBLEM  STATEMENT 


SOURCE(S) 


1. 

Expansion  of  technology  in  the  next  five  years 
will  be  explosive. 

i 

s 

u 

2. 

Terminology  is  inconsistent  in  the  A1  world. 

k  m 

3. 

Systems  only  evolve  gradually  because  it  takes 
a  good  deal  of  experimentation  to  achieve  high 
performance. 

k 

4. 

A1  systems  require  a  continuous  relationship 
with  an  expert. 

t 

5. 

Eventually,  as  a  result  of  A1  systems,  automa¬ 
tion  will  occur  and  many  jobs  will  disappear 
or  change  radically. 

m 

6. 

No  programs  exist  yet  which  can  understand 
simple,  mechanical  causality. 

s 

7. 

AI  systems  are  unable  to  recognize  or  deal 
with  problems  for  which  their  own  knowledge 
is  inapplicable. 

k 

8. 

Al  methods  must  be  augmented  with  conven¬ 
tional  techniques  to  solve  real  problems. 

P 

A. 2  Enviromneiit 
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A. 2  Environment 

The  section  on  environment  examines  tools  and  methodologies  involved  in  the  development  and 
in-service  support  of  computer  software.  Five  major  categories  which  have  surfaced  are  described: 
Disciplined  Methods,  Labor-intensive,  Tools,  Reinvention  and  Capital  Investment. 

A. 2.1  Disciplined  Methods 

Disciplined  methods  refers  to  the  use  of  adequate  engineering  discipline  in  software  activities.  Lack 
of  software  discipline  varies  from  those  who  fail  to  apply  the  required  discipline  to  a  software 
problem,  to  those  who  fail  to  recognize  the  need  for  software  discipline.  The  result  is  lack  of 
enforcement  of  good  programming  standards  which  includes  effective  use  of  structured  programming 
techniques,  inefficient  software  configuration  management,  lack  of  baselining  and  documentation, 
and  an  insufficient  system  engineering  technique  applied  to  a  program.  For  itemised  problems  see 
tables  A. 2. 6-12  and  A. 2. 6-13. 


A.2.2  Labor  Intensive 

Since  software  is  a  labor-intensive  technology,  methods  to  improve  the  efficiency  of  people  are  a 
continuous  concern.  One  means  to  increase  productivity  is  the  automation  of  manual  processes. 
Another  suggested  means,  is  not  only  to  reduce  mechanical  human  activities,  but  to  enhance 
creative  possibilities  through  automated  systems.  For  itemized  problems  see  tables  A.2.6-14  and 

A. 2.6  15. 


A. 2. 3  Tools 

Development  and  support  tools  are  needed  to  improve  productivity  in  design,  coding,  test,  support 
and  management.  Due  to  the  diversity  and  complexity  of  systems,  emphasis  is  placed  on  tools  and 
procedures  with  wide-spread  application.  The  needs  for  automated  software  design  and  support 
tools  include  uniform,  portable,  yet  tailored  tools  with  applications  towards  each  phase  of  the 
software  life  cycle.  Other  desirable  tools  include:  software  documentation  systems;  configuration 
management  systems;  data  base  management  systems;  management  information  systems;  software 
libraries;  and  comprehensive  software  development,  support  and  test  tools  (editors,  code  syntax 
checkers,  scenario  generators,  operational  test  data  reduction  software,  etc.).  For  itemized  problems 
see  tables  A  2.6- 16  and  A. 2. 6-17. 


A.  2. 4  Reinvent  ion 

Reinvent  ion  refers  to  the  inability  of  software  development  to  reuse  functionally  similar  software 
developed  for  other  systems  resulting  in  higher  development  costs.  For  itemized  problems  see  tables 
A. 2.6  18  and  A  2.6-19 
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A.2.S  Capital  Investment 


A.2.5  Capital  Investment 

Capital  investment  is  required  to  solve  many  existing  software  problems.  Yet,  software  is  given  a 
lower  priority  in  funding  by  the  DOD.  The  funds  allocated  for  capital  investment  and  labor  are 
signficantly  lower  than  the  estimated  need.  For  example,  there  is  an  unwillingness  to  make  the 
essential  capital  investment  required  to  provide  better  support  environments  with  an  ultimate  goal 
of  improving  software  engineering  and  eliminating  reinvention.  For  itemized  problems  see  tables 
A.2.6-20  and  A.2.6-21. 

A. 2.6  Environment  Problem  Tables 

The  following  tables  show  the  problems  which  surfaced  in  each  of  the  Environment  categories 
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PROBLEM  STATEMENT 


SOURCEi 


1. 

Retraining  of  individuals  is  costly  so  movement  is  not  pro¬ 
moted. 

f 

2. 

Contractor’s  software  tools  rarely  leave  their  shops  when  the 
product  is  delivered. 

f 

3. 

Lack  of  consistent,  disciplined  methods  impact  Embedded 
Computer  Systems  (ECS)  because  the  software  is  developed 
by  independent  groups. 

f 

4 

Need  for  better  definitions  of  software  terms,  measures  of 
software  qualities,  and  methods  of  measuring  them. 

c  d 

h 

5. 

Differing  policies  exist  among  various  Federal  agencies  and 
industry. 

h 

6. 

Lack  of  standardization  of  Data  Item  Descriptions. 

h 

7. 

Within  DOD,  partial  and  inadequate  standards  exist. 

h 

8. 

Standards  are  rigid  and  lack  the  ability  to  be  tailored. 

b 

h 

9 

Lack  of  well-defined,  consistent  requirements  causes  Software 
Quality  Assurance  difficulties. 

c 

h 

10. 

No  standard  set  of  software  acceptance  criteria  within  DOD. 

b  c 

h 

11 

Government  standards  do  not  recognize  that  differences  in 
program  size  should  influence  the  decision  to  apply  standards. 

h 

12 

Standards  address  too  much  detail  disregarding  the  end  prod¬ 
uct  objectives. 

h 

A. 2.6  Environment  Problem  Tables 
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A. 2.6  Environment  Problem  Tables 


Table  A.2.6-15:  Artificial  Intelligence  Labor  Intensive 
PROBLEM  STATEMENT  ~I  SOURCE(S) 


1.  Acquisition,  organization  and  structuring  of 

1 

knowledge  is  a  manual  process. 

Overlapping  AI  References:  1  (Skills) ,  1  (Availability). 
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Table  A. 2.6-16:  Conventional  Software  Tools 
PROBLEM  STATEMENT  ~ 

Contractor  is  often  forced  to  use  a  government  tool  which 
requires  use  of  an  unfamiliar  system. 

Need  for  standardization  of  high  order  programming  lan¬ 
guages. 

Deviations  in  language  definition  and  implementation  forbids 
transportability  of  High  Order  Language  (HOL)  programs. 

Lack  of  consistent  standards  for  software  causes  problems  in 
acquisition  and  development. 

Effective  and  efficient  maintenance  is  difficult. 

Software  is  often  re-engineered.  a 

Tools  are  unavailable,  unpublicized,  difficult  to  understand  a 
or  use,  or  inefficient 

Inconsistent  computer  support  of  software  development. 
Difficulties  in  transitioning  generic  (non-weapon)  specific 
software  tools  to  weapon  system  programs. 

Non-modular  tools  are: 

•  expensive; 

•  untimely; 

•  difficult  to  maintain; 

•  inflexible  to  change; 

•  unreliable;  and 

•  non-responsive  to  user  requirements. 


ind  In  -I  I  .ii  gri  pim  essors. 

12.  Resource  constraints  only  allow  a  minimal  number  of  tools 
to  be  developed. 

lib  Poor  documentation  of  tools  results  in  a  lower  level  of  quality 
results  of  I  lie  system 

M  Non-uniformity  among  tools. 

Ir>  Strive  for  language  independence  rather  than  support  for  a 
specific  language 

It*  Should  apply  existing  tools. 
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A.2.6  Environment  Problem  Tables 


Table  A. 2. 6-17:  Artificial  Intelligence  Tools 


PROBLEM  STATEMENT 

SOURCE(S) 

1.  Cost  performance  standards  are  not  fully  un¬ 
derstood  yet. 

m 

A. 2. 6  Environment  Problem  Tables 


Table  A. 2.6-18:  Conventional  Software  Reinvention 


PROBLEM  STATEMENT 

SOURCE(S) 

1. 

Designs  and  implementations  of  previous  systems  are  not  cap¬ 
tured  and  reused  in  succeeding  systems. 

abcdefgh 

2. 

A  rigorous  interface  standard  must  be  provided  for  all  candi¬ 
date  reusable  components. 

f 

3 

A  library  of  reusable  components  must  evolve  for  use  by  all 
development  activities. 

f 

4. 

An  index  of  reusable  components  must  be  included. 

b  f 

5 

A  rigorous  design  standard  must  be  provided  for  any  major 
software  system  or  subsystem  to  be  implemented  as  a  skele¬ 

e  f 

ton. 

6. 

A  library  of  subsystem  or  system  skeletons  must  be  provided. 

f 

7. 

A  set  of  supporting  tools  must  be  implemented. 

f 

8. 

Most  application-specific  software  is  developed  new  for  each 
application. 

g 

A.2.6  Environment  Problem  Tables 


Table  A. 2.6-19:  Artificial  Intelligence  Reinvention 


A.2.6  Environment  Problem  Table « 


Table  A  2.6  20:  Conventional  Software  Capita!  Investment 


_ PROBLEM  STATEMENT _ 

1.  Many  DOD  Computer  Support  facilities  are  overloaded  and 
use  aging  equipment. 

2.  Software  is  developed  on  target  hardware  which  is  not  de¬ 
signed  to  support  such  development. 

3.  Lack  of  sufficient  time  is  spent  on  Independent  Research  and 
Development. 

4.  Schedule  slips  occur  resulting  in  cost  and  resource  consump¬ 
tion. 

5.  Major  software  costs  in  development,  operation,  and  mainte¬ 
nance  phases. 
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A. 3  Software  Product 


A. 3  Software  Product 

The  software  product  consists  of  the  operational  embedded  computer  software  as  well  as  the  mate¬ 
rials  necessary  for  life  cycle  support  -  requirement  and  design  specifications,  source  code,  test  data, 
system  generation  data,  unique  support  tools  etc.  The  following  five  categories  briefly  describe 
some  subdivisions  within  the  Software  Product:  Doesn’t  Meet  the  Need;  Software  Metrics;  Design 
Attributes;  Documentation  and  Immutable  Software, 

A. 3.1  Doesn’t  Meet  the  Need 

The  section  refers  to  the  failure  of  the  deployed  system  to  meet  the  user's  need.  One  such  situation 
occurs  when  requirements  are  either  stated  or  implemented  incorrectly  which  results  in  changes  to 
correct  the  problems.  For  itemized  problems  see  tables  A. 3. 6-22  and  A. 3.6-23. 

A. 3. 2  Software  Metrics 

Software  metrics  are  aimed  at  providing  analytic  models  and  empirical  data  on  software  to  aid  in 
the  choice  of  software  engineering  techniques,  estimate  development  resources,  and  evaluate  future 
costs.  For  itemized  problems  see  tables  A. 3.6-24  and  A.3.6-25. 

A. 3. 3  Design  Attributes 

The  Design  provides  an  acceptable  programming  solution  to  the  problems  stipulated  in  the  Require¬ 
ments  Specification.  As  such,  the  design  specification  contains  a  solution  to  the  user’s  problem. 
For  itemized  problems  see  tables  A. 3.6-26  to  A. 3. 6-28. 

A. 3. 4  Documentation 

Documentation  should  convey  information  on  or  about  the  computer  system  developed.  Both 
managerial  and  technical  work  should  be  included  as  well  as  transitional  documentation  helpful  in 
bridging  the  gap  between  phases.  Also,  documentation  serves  as  a  baseline  from  which  changes 
or  upgrades  a~e  made.  Although  documentation  is  an  important  aspect  in  the  development  of 
computer  systems,  the  resources  allocated  do  not  reflect  the  actual  costs  necessary  to  produce 
adequate  documental  n.  For  itemized  problems  see  tables  A. 3. 6-29  and  A. 3. 6-30. 

A. 3. 5  Immutable  Software 

This  section  refers  to  software  packages  that  are  system  unique,  non-portable  and  non-reusable. 
Soft  ware  developed  for  embedded  computer  systems  is  generally  tailored  to  the  specific  application 
and  hardware  environment,  without  regard  for  reusability.  Consequently,  acquisition  agencies  are 
forced  to  pay  over  and  over  again  for  software  that  has  essentially  been  written  elsewhere.  For 
itemized  problems  see  tables  A. 3. 6-31  and  A. 3. 6-32. 
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A. 3.6  Software  Product  Problem  Tables 


A. 3.6  Software  Product  Problem  Tables 

The  following  tables  show  the  problems  which  surfaced  in  each  of  the  Software  Product  categories. 


A. 3.6  Software  Product  Problem  Tables 


Table  A. 3.6-22:  Conventional  Software  Doesn’t  Meet  the  Need 


PROBLEM  STATEMENT 

SOURCE(S) 

1.  Software  is  expected  to  solve  hardware  problems. 

b  f 

2.  Software  is  difficult  to  develop  and  support,  understand  com- 

a  b  c  d  e  f 

pletely.  and  measure  its  performance  fairly. 

3  Inadequate  communication  in  system. 

c  f 

4.  Ambiguous,  unclear,  and  incomplete  requirements  definit  ion 

b  c  f 

5.  Necessary  to  define  completion  and  performance  criteria 

r 

6.  Incompatibilities  between  test  equipment,  at.  different  levels 

t 

of  maintenance 

7  7  V 
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A. 3.6  Software  Product  Problem  Tables 


Table  A.3.6-23:  Artificial  Intelligence  Doesn’t  Meet  the  Need 
PROBLEM  STATEMENT  SOURCE(S) 

1.  Myths  regarding  human  and  computer  intelli-  j 
gence  may  mislead  the  user  in  terms  of  system 
capabilities. 

Overlapping  AI  References:  6(Requirements),  9(  Requirements),  l(Product  Assurance). 
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A. 3.6  Software  Product  Problem  Tables 


Table  A. 3.6-24:  Conventional  Software  Metrics 


PROBLEM  STATEMENT 

SOURCE(S) 

1.  Lack  of  good  analytical  methods  and  hard  empirical  data 

a  b  c  d  e  f 

needed  to  estimate  future  costs  and  mission  impacts. 

2.  No  validated  models  of  life  cycle  costs  and  productivities  in 

a  b  c  e  f 

development  and  support. 

3.  Unexpected  changes  in  hardwired  software  during  software 

f 

project  life  cycle. 

4.  Undecided  as  to  whether  or  not  Configuration  Item  (Cl)  or 

e  f 

Computer  Program  Configuration  Item  (CPCI)  standards 

should  be  applied  to  firmware. 

5.  Need  to  separate  and  conduct  measurement  of  creativity  and 

f 

implementation. 

6.  Contractors  are  not  requested  to  report  detailed  software 

a 

data  on  any  aspect  of  software  acquisition  or  support. 

7.  Managers  need  regularly  scheduled  updates  to  show  cost  and 

d  e 

schedule  states. 

A. 3.6  Software  Product  Problem  Tables 


Table  A. 3.6-25:  Artificial  Intelligence  Metrics 


PROBLEM  STATEMENT 

SOl)RCE(S) 

1.  Evaluation  of  an  Al  project  is  difficult. 

q 

U 

2.  There  exists  no  true  scale  of  measure  to  tell 

q 

how  high  a  rating  that  a  system  can  realisti- 

cally  achieve. 

Overlapping  AI  References:  4(Management),  1  (Acquisition),  2(Acquistion). 
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PROBLEM  STATEMENT 


SOUHCE(S) 


1 

Need  a  more  focused  body  of  literature  for  software  design 
as  it  relates  to  system  computational  architecture  and  per¬ 
formance. 

a 

f 

2. 

Lack  of  an  understanding  of  both  software  design  implications 
and  hardware  architecture. 

b 

r 

3. 

Need  for  safety  features  to  prevent  loss  of  security  informa¬ 
tion. 

f 

4. 

Design  requirements  do  not  exist  to  ensure  the  best  user  sys¬ 
tem  interface. 

b 

f 

5. 

Misunderstandings  occur  between  software  and  hardware  en- 

f 

gineers. 

6. 

Software  engineers  develop  misconceptions  of  how  hardware 
performs. 

b 

r 

R 

7. 

Inadequately  designed  requirements. 

b 

c  f 

8 

8. 

Incorrect  assumptions  made  by  software  engineers. 

f 

R 

9. 

Software  often  lacks  modularity  resulting  in  memory  waste. 

b 

f 

10. 

Improper  selection  of  a  software  language  to  be  used. 

b 

f 

11. 

Programming  style  contributes  to  faulty  design 

b 

f 

12 

Poor  documentation  and  conformity  to  a  standard. 

a 

f 

13 

Software  is  often  written  to  fit  the  outdated  hardware  system 
used 

a 

f 

14. 

Too  ambiguous  in  defining  objectives. 

f 

15. 

Monolithic  development  leads  to  cost  overruns  and  schedule 
delays. 

f 

16. 

The  software  life  cycle  is  not  really  considered  when  design 
decisions  are  made. 

f 

17 

Unclear  understanding  of  design  options. 

c 

g 

18. 

It  is  often  the  case  that  in  order  to  include  a  new  capability 
in  a  system  a  major  software  redesign  is  required. 

a 

b 

c  d  e  f 

g  *> 

19. 

Premature  programming  begins  causing  long-term  difficul¬ 
ties. 

b 

20 

Bottom-up  design  is  implemented  rather  than  top-down. 

b 

21 

Long,  costly  design  phase. 

b 

A. 3.6  Software  Product  Problem  Tables 


Table  A. 3.6-27:  Conventional  Software  Design  Attributes  (Coni.) 


PROBLEM  STATEMENT 

SOUItt’K(S) 

22.  Lack  of  software  transferability  from  one  system  to  another. 

23.  Software  does  not  have  the  same  degree  of  visibility  or  atten¬ 
tion  as  hardware  does. 

24.  The  major  elements  of  weapons  system  software  are  often  not 
integral  with  the  operational  components. 

25.  Weapons  system  software  does  not  fit  previously  defined  pro¬ 
curement  categories. 

26.  Need  to  allocate  software  resources  differently  than  hardware 
portions  of  a  system  program. 

27.  Known  risk  reduction  techniques  need  to  be  employed  for 
software. 

28.  Need  simplicity  of  interfaces  among  Cl’s  and  CPCI’s. 

bed 

c 

c 

c 

c 

c 

e 
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A. 3.6  Software  Product  Problem  Tables 


Table  A. 3.6-28:  Artificial  Intelligence  Design  Attributes 


PROBLEM  STATEMENT 

SOURCE(S) 

1.  Problems  arise  in  making  associative  data 

i 

bases  (which  require  functions  for  adding,  re- 

moving  and  fetching  data  items)  efficient  and 

meaningful. 

2.  Al  systems  are  not  noted  for  their  speed. 

111  o  s  l 

3  In  Al  programs,  data  structures  tend  to  be 

II 

large  and  complex. 

4  Improving  and  defining  descriptions  often  mo- 

j 

bilizes  powerful  constraints  that  force  conclu- 

sions  directly  without  sophisticated  problem 

solving  and  reasoning  mechanisms 

5  Simple  search  techniques  cannot  find  optimal 

j 

paths  and  may  be  inefficient. 

6.  Difficulties  arise  when  the  expert  attempts  to 

p 

map  his  explanations  directly  into  the  formal- 

ism 

7.  There  often  exists  voluminous  amounts  of  in- 

i  r  s  t  u 

formation  in  the  definition  phase  of  an  expert 

system. 

8.  Experts  often  trim  their  knowledge  to  fit  the 

t 

knowledge  structure  more  conveniently. 

9.  Knowledge  bases  exist  but  little  help  is  avail- 

i  k 

able  for  initial  design  decisions. 

10.  Because  knowledge  systems  are  not  hierarchi- 

o 

cal,  they  are  difficult  to  decompose. 

J 
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A. 3.6  Software  Product  Problem  Tables 


Table  A. 3.6-29:  Conventional  Software  Documentation 


PROBLEM  STATEMENT 

SOURCE(S) 

1.  Documentation  effort  is  relaxed  when  schedule  slips  occur. 

a  b  d  f 

2.  Documentation  is  often  deleted  if  funding  is  reduced. 

a  b  d  f 

3.  There  exists  inconsistencies  as  to  what  documentation  is  nec- 

a  d  e  f 

essary. 

4.  Errors  occur  in  the  actual  production  of  documents. 

a  f 

5.  There  exists  little  or  no  documentation  in  key  areas  of  design 

a  b  d  f 

and  analysis. 

6.  Inadequate  documentation  exists  resulting  in  a  lack  of  com- 

a  f 

plete  project  history. 

7.  Requirements  of  information  scope,  depth,  and  format  are 

a 

usually  ignored. 

8.  In  contracting  for  software  documentation,  there  exist  prob- 

a  e 

lems  in  knowing  what  to  ask  for. 

9.  No  standard  documentation  Data  Item  Description’s  (DID’s) 

a 

for  software. 

10.  Factors  of  difficulty  with  regard  to  documentation: 

e 

•  system  complexity; 

•  project  resources; 

•  development  stage;  and 

•  documentation  overlap. 

1 1.  System  interfacing  must  appear  in  documentation. 

e 

12.  Need  sufficient  manpower  to  manage/review  the  documcnta- 

d  e 

tion. 

13.  Anticipated  maintenance  requirements  must  be  developed. 

d  e 

14.  TYaceability  between  documents  must  exist. 

e 

If*.  Problem  in  defining  what  comprises  a  hardware  intensive  ver- 

e 

sus  software  intensive  application. 

A. 3. 6  Soft  whip  Product  Problem  Tables 


Table  A. 3. 6 -30:  Artificial  Intelligence  Documentation 

problemstatement  soimc:K(sj" 


99.  None  Noted 
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A. 3.6  Software  Product  Problem  Tables 


Table  A. 3. 6-31:  Conventional  Software  Immutable  Software 


_ PROBLEM  STATEMENT _ 

No  formal  approach  exists  to  exploit  the  potential  of  reactions 
to  adversary  threats  within  systems. 

No  emphasis  on  systems  to  meet  changes  in  enemy  defensive 
techniques. 

Designers  do  not  optimize  available  computer  resources. 

Lack  of  standard  hardware  used. 

Inadequate  transfer  of  information  regarding  existing  soft¬ 
ware  resources. 

Specifications  are  not  written  in  a  common  language  or  for¬ 
mat. 

Software  is  not  documented  to  be  reused. 

Lack  of  software  development  standards  exist. 
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A. 3. 6  Software  Product  Problem  Tables 


Table  A. 3. 6-32:  Artificial  Intelligence  Immutable  Software 


PROBLEM  STATEMENT 

SOURCE(S) 

99  None  Noted 

Overlapping  AI  References:  4(Management),  l(Acquisition) 
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A. 4  People 


A. 4  People 

A  shortage  of  skilled  system  engineers,  software  engineers  and  managers  has  sin  lare.l  as  a  result 
of  the  rapid  spread  of  digital  technology.  Consequently,  three  problematic  areas  have  manifested 
themselves  in  professional  skills,  professional  availability,  and  professional  incentive  which  are  ex¬ 
amined  as  separate  categories  in  this  section. 

A.4.1  Skills 

One  of  the  most  highly  valued  and  respected  skills  of  an  engineer  is  the  ability  to  properly  prepare 
software  requirements,  designs,  and  to  provide  adequate  support  for  future  modifications  to  systems 
For  itemized  problems  see  tables  A.4,4-33  and  A. 4. 4-34. 

A.4.2  Availability 

The  availability  of  qualified  professionals  for  software  development  does  not  meet  the  need.  For 
itemized  problems  see  tables  A. 4.4-35  and  A.4.4-36. 

A. 4. 3  incentive 

Incentive  refers  to  the  idea  that  the  software  engineering  career  field  should  recognize  and  encourage 
skilled  professionals  through  incentives  such  as  better  working  conditions,  educational  opportuni¬ 
ties,  promotions  and  salary  increases.  Along  with  this  concept  is  the  idea  that  professionals  should 
be  trained  in  managerial  and  technical  skills.  For  itemized  problems  see  tables  A. 4.4-  37  and  A  4.4 
38. 


A. 4. 4  People  Problem  Tables 

The  following  tables  show  the  problems  which  surfaced  in  each  of  the  People  categories. 


A. 4.4  People  Problem  Tables 


Table  A. 4. 4-33:  Conventional  Software  Skills 
PROBLEM  STATEMENT 

1 .  Short  age  of  skilled  systems  engineers,  software  engineers,  and  a 
managers. 

2  Individuals  must  require  a  broad  range  of  knowledge  in  many 
areas. 

3.  Communication  techniques  are  often  ineffective. 

4.  Managers  often  lack  technical  knowledge.  I 

5.  No  formal  instruction  adequate  to  develop  needed  skills. 

6  A  few  scarce  experts  must  guide  an  entire  software  project. 
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A. 4.4  People  Problem  Tables 


Table  A.4.4-34:  Artificial  Intelligence  Skills 


PROBLEM  STATEMENT 

SOIMK'K(S) 

1.  Need  for  multiple  experienced  managers  and 
engineers  in  the  field  of  Al. 

n 

Overlapping  AI  References:  4(Requirements),  6(Requirements),  3(Management),  l(Product 
Assurance),  1  (Availability). 
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A. 4. 4  People  Problem  Tables 


1. 

2. 

3. 

4 


Table  A. 4. 4-35:  Conventional  Software  Availability 


PROBLEM  STATEMENT 


Greater  demand  for  engineers  and  computer  scientists  than 
individuals  actually  exist. 

Number  of  people  actually  needed  on  a  software  project  is 
not  always  clear. 

Lack  of  experienced  technical  personnel  within  the  Navy. 
Severe  personnel  shortage. 
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f 

f 

f 

r 


A. 4.4  People  Problem  Tables 


Table  A. 4.4-36:  Artificial  Intelligence  Availability 


SOURCE(S) 


PROBLEM  STATEMENT 


There  only  exists  a  limited  number  of  engi 
neers  in  AI  software  development. 


A. 4. 4  People  Problem  Tables 


Table  A.4.4-37:  Conventional  Software  Incentive 


_ PROBLEM  STATEMENT _ 

Sellers  market  exists;  high  rate  of  turnover. 

Excellent  technical  people  are  promoted  out  of  their  fields 
into  management  positions. 

Lack  of  reward  for  excellence. 

Industry  outbids  the  Government  for  desirable  candidates. 
Standard  three  year  tour  of  duty  policy  exists  for  Air  Force 
officers. 


SOURCE(S) 
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Appendix  B 
Questionnaire 


ARTIFICIAL  INTELLIGENCE  SOFTWARE  DEVELOPMENT 
CASE  STUDY  QUESTIONNAIRE 


The  attached  questionnaire  has  been  sent  to  you  by  the  Software  Systems  Engineering  Directorate 
(SSE)  of  Sanders  Associates  Inc.,  a  large  defense  electronics  firm  located  in  southern  New  Hamp¬ 
shire.  SSE  provides  software  support  to  the  users  of  Sanders  computing  facilities  and  undertakes  a 
broad  spectrum  of  software  engineering  projects. 

One  such  project  requires  SSE,  under  contract  to  Rome  Air  Development  Center' ,  to  perform  a 
study  on  the  acquisition,  management,  and  control  of  AI  software.  While  the  Department  of  Defense 
has  established  numerous  standards,  policies,  and  guidelines  for  the  acquisition  and  development 
of  many  types  of  conventional  software,  no  such  information  exists  for  AI  software.  The  primary 
purpose  of  this  study  is  to  postulate  a  model  for  the  development  of  AI  software  which  could 
subsequently  be  used  to  devise  associated  policy,  standards,  and  guidelines.  One  way  we  propose  to 
address  our  contract  requirements,  is  to  perform  a  number  of  case  studies  on  AI  system  development 
In  this  way,  we  can  attempt  to  ferret  out  common  methodologies  and  pay  particular  attention  to 
those  techniques  that  are  considered  to  be  successful.  Consequently,  the  objective  of  the  attached 
questionnaire  is  to  obtain  data  from  sources  active  in  the  AI  arena  to  support  the  development  of 
a  viable  model. 

The  questionnaire  is  divided  into  four  parts.  The  first  part  provides  an  introduction  to  the  overall 
questionnaire  by  discussing  the  concept  of  a  software  development  model.  One  model  typically 
used  by  the  Department  of  Defense  is  presented  pictoriaily  and  discussed  briefly.  The  second  part 
solicits  information  of  a  background  nature  that  may  be  used  to  weigh,  at  least  in  a  qualitative 
sense,  responses  to  the  questions  in  the  remaining  two  parts.  The  third  part  is  devoted  to  technical 
issues  regarding  the  overall  construction  and  support  of  the  system.  This  part  is  further  divided 
into  five  sections.  The  first  section  contains  general  questions  pertinent  to  modeling  AI  software 
development.  The  remaining  four  sections  parallel,  in  a  broad  sense,  major  activities  associated 
with  conventional  software  development  and  field  support  that,  we  assume,  have  counterparts  in  AI 
software  development.  Finally,  the  fourth  part  contains  miscellaneous  questions  whose  placement 
in  one  of  the  sections  of  Part  III  would  have  been  inappropriately  narrow. 

It  is  intended  that  the  response  to  the  questionnaire  be  devoted  to  experiences  encountered  with 
one  AI  system.  If  you’ve  had  experience  with  more  than  one  system,  perhaps  you  could  choose  the 
one  that  you  think  best  describes  the  AI  software  development  process.  We  suspect  that  when  you 
review  the  questionnaire  you  will  find  that  many  questions  can  be  answered  quite  briefly  while  others 
will  require  more  lengthy  responses.  In  order  to  reduce  the  amount  of  time  you  might  spend  in 

1  Contract  No.  F30602-85-C-0254,  Technical  Contact:  Richard  Evan*,  315-330-3528 
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Appendix  B  Questionnaire 

responding  to  the  more  demanding  questions,  you  may  attach  or  reference  available  documentation 
where  appropriate.  We  would  appreciate  any  references  be  as  specific  as  possible. 

We  greatly  appreciate  your  important  input  to  this  effort  and  will  gladly  acknowledge  your  contri¬ 
bution  in  our  Final  Report.  Furthermore,  we  will  be  happy  to  share  any  nonconfidential  information 
that  we  collect  from  other  sources. 

If  you  have  any  questions  or  comments  concerning  this  questionnaire,  please  fool  free  to  contact  us 
as  indicated  below. 

Again,  thank  yon  for  your  cooperation. 
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Sandy  King 

Sr.  Software  Engineer 

603-885-9242 


Larry  Fry 
Study  Director 
603-885-9208 

Sanders  Associates 

Software  Systems  Engineering 

MER24-1283 

95  Canal  Street 

Nashua,  NH  03061 
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Appendix  D  Questionnaire 


PART  I:  INTRODUCTION 

Systems  developed  for  the  Department  of  Defense  (DOD)  are  becoming  increasingly  complex, 
requiring  more  time  and  a  larger  budget  to  complete.  From  a  historical  standpoint,  DOD  experience 
indicates  a  number  of  cases  where  the  acquisition  of  software  has  exceeded  the  predetermined  cost 
and  schedule.  To  assure  a  higher  success  rate  in  acquiring  systems,  the  DOD  has  set  several 
objectives.  Principle  objectives  among  them  are  that  the  contracted  system: 

•  satisfy  the  needs  defined  by  the  DOD,  and 

•  be  developed  within  the  allocated  budget  and  schedule. 

Consequently,  the  DOD  has  mandated  that  contractors  adhere  to  a  methodical  engineering  ap¬ 
proach  in  order  to  provide  visibility  and  control  over  the  development  process. 

The  engineering  approach  mandated  by  the  DOD  requires  various  levels  of  system  and  software 
definition  to  occur  over  time.  At  the  completion  of  each  level  of  definition  the  products  (i.e. 
documentation  or  code)  associated  with  that  level  are  frozen  in  a  state  such  that  any  future  changes 
must  be  treated  in  a  formal  manner.  These  baselines,  as  the  DOD  calls  them,  consist  of  functional, 
allocated,  and  product  baselines  (see  Figure  1).  The  functional  baseline  defines  requirements  for 
the  overall  system.  The  allocated  baseline  freezes  software  requirements  and  the  product  baseline 
reflects  the  “as  built”  design.  A  formal  review,  attended  by  representatives  of  the  government  and 
the  contractor,  is  the  vehicle  for  ascertaining  readiness  for  establishing  a  baseline.  Once  established, 
these  baselines  are  under  government  control  and  any  changes  thereto  must  be  agreed  to  by  the 
government  prior  to  implementation.  Representative  of  the  controlled  baseline  entity  is  a  document 
in  which  information  appropriate  to  the  baseline  is  recorded. 

In  addition  to  the  government  controlled  baselines,  the  most  recently  issued  DOD  software  de¬ 
velopment  standard,  DOD-STD-2167,  also  requires  that  the  developer,  through  the  developmental 
configuration,  control  the  design  and  the  code  as  it  evolves.  Inherent  in  this  stipulation  is  the 
requirement  to  conduct  numerous  internal  reviews  which  are  intended  to  act  as  checkpoints  for 
determining  development  progress. 

Because  DOD  systems  exist  in  a  constantly  changing  environment,  planning  and  designing  for 
change  is  a  high  priority  requirement  within  the  DOD.  Standards  such  as  DOD-STD-2167  address 
this  requirement  by  stipulating  analysis,  design,  and  coding  standards  to  be  adhered  to  by  contrac¬ 
tors.  These  standards  are  aimed  at  producing  software  that,  when  fielded,  can  be  easily  maintained 
and  modified. 

Because  of  its  research  oriented  nature,  the  development  of  A1  software  has  not  reached  t  lie  maturity 
of  process/product  standardization  that  conventional  software  has  achieved.  The  DOD  recognizes 
the  importance  and  value  of  Al  software  and  is  concerned  that  its  development  be  managed  and 
controlled  in  a  manner  which  will  provide  visibility  into  its  development.  From  the  perspective  of 
the  DOD,  the  desire  to  acquire  a  high  quality,  maintainable  product  applies  equally  to  A I  software 
as  it  does  to  conventional  software. 

With  this  background  in  mind,  the  questionnaire  is  oriented  towards  obtaining  data  that  will 
highlight  the  activities  involved  in  AI  software  development,  the  approach  taken,  and,  just  as  im¬ 
portantly,  the  management  visibility  and  control  over  the  product  as  it  evolves.  Although  the 
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Baselines: 


Function*!  Baseline 


Allocated  Baseline 


Review*: 


Phases: 


Products. 


Require  menu 
Review 


Software  _ 

Requirement* 

Analysis  Preliminary 

_ _  Deeifn 


I  Developmental  Configuration 


Design 

Review 


Test 

Readme*! 

Review 


Product  Baseline 


1  CooJtguration 
Audita 
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Coding  and 
Unit  Testing 


Software 
Integration 
and  Tasting 


'  Software 
Performance 
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Figure  B-l:  Software  Development  Cycle 
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Appendix  B  Questionnaire 


process  of  AI  software  development  may  vary  significantly  from  the  process  presented  in  conven¬ 
tional  software  development  models,  the  need  for  managerial  insight  must  be  satisfied.  For  instance, 
systems  which  take  three  to  four  years  to  develop  must  be  measurable  in  terms  of  progress  and 
expectations.  To  this  end,  we  hope  the  questions  will  provide  the  data  to  develop  models  which 
accurately  represent  the  technological  and  management  issues  involved  in  AI  software  development. 


Appendix  B  Questionnaire 


PART  II:  BACKGROUND 

The  purpose  of  this  part  is  to  identify  basic  introductory  information  about  the  subject  system. 
In  addition,  certain  information  is  requested  to  help  put  in  proper  perspective  the  responses  to 
engineering  and  management  related  questions  contained  in  Part  III  of  this  questionnaire. 

Bl.  Please  identify  the  name  of  the  system  to  which  the  responses  to  this  questionnaire  are 
applicable.  Briefly  describe  the  purpose  and  nature  of  the  system  and  how  someone  may  use  the 
system  to  solve  problems  relevant  to  the  system  purpose. 

B2.  Was  the  source(s)  of  funding  for  the  project  internal,  contractual,  or  a  grant?  If  other,  please 
specify. 

B3.  Was  this  the  first  AI  project  that  your  company  was  involved  in?  If  not,  how  numy  systems 
did  the  company  develop  prior  to  this  one? 

B4.  How  many  engineers  were  assigned  to  the  project?  Of  these,  how  many  had  AI  development 
experience  prior  to  this  project? 

B5.  How  long  did  it  take  to  develop  the  system  (man-months)  from  problem  definition  to  first 
prototype?  How  much  longer  (after  the  first  prototype)  did  it  take  to  field  the  system?  Was  the 
schedule  predefined  either  by  management,  the  customer  or  the  amount  of  funding  allocated  for 
the  entire  project? 

B6.  How  many  engineers  assigned  to  the  project  had  prior  experience  in  the  development  of 
conventional  software  for  the  Department  of  Defense?  Were  military  standards  dictated  on  any 
of  these  projects?  To  what  extent  would  you  say  that  military  standards,  or  any  other  standards 
governing  the  development  of  conventional  software  (please  identify),  influenced  the  development 

of  the  AI  system? 

B7.  The  success  of  a  system  can  be  measured  by  many  criteria.  Some  criteria  may  be: 

•  demonstrating  the  feasibility  of  a  new  technology, 

•  overall  user  satisfaction, 

•  significant  productivity  gains 

To  what  degree  do  you  consider  the  system  a  success  for  each  of  the  above  criteria?  For  those 
criteria  for  which  you  weren’t  successful,  is  there  something  you  would  do  differently  in  your  next 
endeavor  to  improve  your  overall  success?  If  so,  what  would  that  be?  Are  there  any  other  criteria 
against  which  you  would  measure  the  success  of  the  system?  If  so,  what  are  they  and  how  did  your 
system  fare  against  these  criteria? 
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PART  m>  DEVELOPMENT  CYCLE 


Tha  purpose  of  this  part  is  to  determine  the  detailed  technical  and  management  issues  that  influ¬ 
enced  the  scope,  development,  and  acceptance  or  certification  of  the  system  This  pari  is  divided 
into  five  sections:  general  procedure,  requirements  definition/analysis,  system  construction,  system 
evaluation/ validation  and  field  support.  Note  that  the  division  is  arbitrary  and  does  not  imply 
that  AI  software  should  necessarily  be  developed  in  a  framework  consisting  of  five  corresponding 
phases.  However,  the  phases  do  parallel  major  objectives  in  developing  good  conventional  software 
and  may  have  some  applicability,  in  a  broad  sense,  to  the  development  of  good  A I  software. 


GENERAL  PROCEDURE 

GP1.  A  model  for  AI  software  development  could  consist  of  the  following  stages. 

•  Identification  -  determining  problem  characteristics 

•  Conceptualization  -  finding  concepts  to  represent  knowledge 

•  Formalization  -  designing  structures  to  organize  knowledge  (tool  building) 

•  Implementation  •  capturing  knowledge 

•  Testing  -  validating  a  knowledge  base  and  system  behavior 

Please  comment  on  the  suitability  of  the  above  stages  for  a  high  level  (  i.e.,  general)  A I  software 
development  model.  Also,  please  contrast  the  above  steps  with  the  approach  or  steps  that  were 
followed  in  the  development  of  the  subject  system.  Indicate  whether  the  steps  were  followed  in  a 
sequential  manner  or  iterated  upon  some  number  of  times.  Lastly,  please  provide  a  brief  description 
of  any  products  (i.e.  documentation  or  code)  generated  as  the  result  of  each  step 

GP2.  Conventional  software  development  efforts  typically  include  review  sessions,  such  as  design 
reviews  and  code  walkthroughs,  to  track  interim  project  progress.  These  review  sessions  may 
be  formal  (customer  attended)  or  informal  (internal  staff  only).  If  applicable,  please  describe  any 
review  sessions  held  and/or  any  documentation  produced  during  the  development  elTort  to  ascertain 
progress. 


REQUIREMENTS  DEFINITION/ANALYSIS 

If  AI.  Conventional  software  development  efforts  typically  begin  with  a  requirements  analysis  phase 
to  identify  and  bound  the  problem  at  hand.  Did  you  perform  a  requirements  analysis  for  t  he  subject 
system? 

UA2.  If  the  answer  to  ItAl  is  “yes”,  proceed  to  It  A3.  If  not,  how  did  you  bound  the  goals  of  the 
development  effort? 

HAT  If  the  answer  to  ItAl  is  “no”,  proceed  to  the  next  section.  Otherwise,  describe  the  activities 
associated  with  the  requirements  analysis  phase. 
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1  Were  users  included  in  the  requirements  analysis  process?  If  so,  to  what  extent  was  their 
input  useful  in  defining  the  system? 

2.  Were  any  software  tools  used  in  the  requirements  analysis? 

3.  If  documentation  was  generated,  describe  it.  Was  the  documentation  reviewed  for  approval? 

4.  Were  the  requirements  frozen  at  the  end  of  the  requirements  analysis  such  that  any  future 
changes  had  to  go  through  an  approval  process? 


SYSTEM  CONSTRUCTION 

SCI.  In  formulating  a  knowledge  base,  the  knowledge  acquisition  process  may  be  exercised  in  a 
number  of  ways.  For  example,  one  approach  is  for  a  knowledge  engineer  to  study  the  domain  area 
and  then  interrogate  an  expert.  The  interrogation  may  be  tape  recorded  to  provide  a  verbatim 
transcript  for  further  reference  and  review.  The  information  obtained  in  the  interview  m&,  be 
assembled  and  ordered  into  subcategories  from  which  a  knowledge  representation  approach  can  be 
determined.  Describe  the  knowledge  acquisition  process  for  your  project.  Include  whether  or  not 
the  process  was  documented.  If  so,  was  the  documentation  reviewed  for  approval?  Lastly,  what 
determined  completion  of  the  knowledge  acquisition  process? 

SC2.  Some  methods  of  knowledge  representation  include  conceptual  dependencies,  semantic  net¬ 
works,  frame  based,  logic  based,  and  production  rule  structures.  What  method(s)  was  selected  for 
your  system?  If  applicable,  how  was  the  method  reviewed  and  approved?  Also,  please  discuss  the 
means  employed  to  accommodate  ill-defined  or  incomplete  knowledge. 

SC3.  In  a  rapidly  developing  discipline  the  domain  knowledge  may  be  continually  increasing.  Since 
initially  formulated,  has  the  size  of  your  knowledge  base  (number  of  rules,  frames,  etc.)  expanded? 
If  it  has,  was  it  necessary  to  obtain  approval  for  each  addition  to  the  knowledge  base?  If  so, 
describe  the  approval  mechanism.  Lastly,  was  there  a  point  at  which  it  was  decided  not  to  include 
any  additional  knowledge  in  the  knowledge  base,  and  was  this  a  managerial  or  a  technical  decision? 

SC4.  The  type  of  knowledge  representation  method  selected  often  determines  the  programming 
tools  used.  What  tool(s)  was  selected  for  inclusion  in  the  subject  system?  If  applicable,  include 
a  description  of  conventional  languages  and  components  used  and  how  they  fit  into  the  overall 

system. 

SC!>.  (Expert  Systems  Only)  What  dimensions  were  considered  in  identifying  the  “expert”  behind 
the  expert  system?  Was  there  any  attempt  made  to  verify  the  accuracy  of  his  or  her  input  prior 
to  the  system  validation  phase? 

SC6  (Expert  Systems  Only)  In  developing  a  knowledge  base  for  an  expert  system,  input  from 
more  than  one  expert  may  result  in  a  mixed  approach  that  is  not  representative  of  any  individual 
expert  How  many  experts  had  input  to  your  knowledge  base?  If  more  than  one,  how  was  the 
information  handled  to  eliminate  conflicting  and/or  redundant  facts  and  approaches? 

SC7  Were  users  introduced  into  the  development  process?  If  so,  to  what  extent  did  user  partici¬ 
pation  influence  the  evolution  of  the  developing  system? 
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SC8.  Exploratory  programming  is  the  conscious  intertwining  of  system  design  and  implement  ation. 
Namely,  as  the  system  is  implemented,  design  changes  may  be  warranted.  Did  your  system  go 
through  an  iterative  period  of  design  and  implementation  changes?  If  so,  what  was  the  magnitude 
of  the  changes  and  was  it  necessary  to  obtain  approval  prior  to  implementing  each  change?  If  so, 
describe  the  review/approval  process. 

SC9.  Based  on  an  accepted  design,  rapid  prototyping  is  an  approach  used  to  quickly  develop  a 
working  system  upon  which  to  build.  Was  this  approach  used  on  the  subject  system?  If  so,  describe 
the  process.  If  applicable,  at  what  points  were  the  evolving  prototypes  subject  to  review?  Once 
approved,  was  each  prototype  frozen  as  a  reference  point  from  which  successive  changes  were  made? 

SC10.  Were  there  any  controls  placed  on  the  software  to  track  updated  versions  generated  during 
system  development?  If  so,  describe  the  control  mechanism. 

SC11.  A  “good”  AI  system  architecture  may  have  the  following  characteristics. 

•  separate  inference  engine  from  knowledge  base 

•  as  uniform  a  knowledge  representation  as  possible 

•  as  simple  an  inference  engine  as  possible 

•  overlapping  knowledge 

Describe  the  characteristics  of  the  subject  system.  To  what  extent  were  the  architecture  character¬ 
istics  of  the  subject  system  influenced  by  the  following  domain  characteristics: 

•  size  of  solution  space 

•  reliability  of  data  or  knowledge 

•  nature  of  data  (dynamic  vs  static) 

•  nature  of  knowledge 

•  user  interfa  .es 

SC12.  The  following  elements  may  be  potential  components  of  an  ideal  AI  system  (brief  descriptions 
of  each  element  are  presented  in  Attachment  1): 

•  Language  Processor 

•  Knowledge  Base 

-  Facts 

-  Rules 


•  Justifier 
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•  Interpreter 

•  Scheduler 

•  Consistency  Enforcer 

•  Communications  Handler 


Which  components  does  the  subject  system  contain?  What  factors  (e.g.  design  constraints,  do¬ 
main  characteristics,  cost/schedule  constraints,  etc)  influenced  the  decision  to  include  or  exclude 
components?  Was  the  architecture  subject  to  review  and  approval  by  managerial  and/or  technical 
associates? 

SC13.  Please  give  an  indication  of  the  size  of  the  AI  system  (number  of  rules  in  the  knowledge 
base,  lines  of  code  or  some  other  easily  understood  unit  of  measure).  Identify  the  division  between 
AI  and  conventional  software  components  in  addition  to  the  percentage  of  code  dedicated  to  each 
of  the  components  listed  below.  Add  components,  if  applicable  to  the  subject  system. 


•  knowledge  base 

•  inference  engine 

•  user  interface 

•  support  environment 

SC14.  Software  tools  are  often  imperative  in  AI  system  development  because  a  small  team  is 
responsible  for  generating  a  large  amount  of  code.  Describe  the  software  development  environment 
under  which  the  subject  system  was  developed.  If  tools  were  not  available,  how  do  you  think  the 
lack  of  same  hindered  the  project? 

SC  15.  How  was  the  AI  software  interfaced  with  conventional  software?  What  problems  were  noted 
in  making  the  interface? 


SYSTEM  EVALUATION/VALIDATION 

SV1.  Please  describe  your  overall  approach  to  evaluating  the  performance  of  the  subject  system. 
What  type  of  test  documents  (e.g.  plans,  procedures,  reports),  if  any,  were  written?  Were  users 
involved  in  the  test  process?  Were  test  strategies  applicable  to  conventional  software  such  as 
multiple  test  levels  (e.g.  module  test,  integration  test,  program  test),  branch  coverage,  boundary 
value  testing,  stress  testing,  etc,  used  to  test  the  AI  system?  If  not,  describe  the  strategy,  if  any, 
used  to  test  the  system. 

SV2.  If  the  knowledge  base,  inference  engine,  and/or  other  components  exist  as  discrete  entities 
in  the  subject  system,  were  they  tested  separately?  How  did  other  architectural  characteristics 
influence  the  test  process? 
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SV3.  Is  the  subject  system  self-modifying  (is  it  capable  of  changing  static  evaluators,  internal  con¬ 
cept  formulations,  or  modifying  its  inference  engine  in  response  to  new  data  or  incorrect  reasoning)? 
If  so,  what  impact  did  this  characteristic  have  on  your  approach  to  testing?  Did  you  repeat  tests 
that  performed  properly  prior  to  the  self-modification?  If  not,  how  did  you  know  that  the  modified 
software  did  not  impact  the  performance  of  the  system  in  a  negative  way? 


SV4.  It  has  been  advocated  that  the  evaluation  of  an  AI  system  should  emphasize  quality  assess¬ 
ment  (ie  overall  user  satisfaction)  rather  than  more  traditional  performance  measurements.  In  one 
particular  investigation,  an  interactive  evaluation  subsystem  wac  built  into  an  expert  system  known 
as  the  Automated  Academic  Advisor^.  The  purpose  of  the  subsystem  was  to  monitor  various  pa¬ 
rameters  of  system  operation  and  to  administer  interactively  questionnaires  to  the  user  while  the 
application  system  was  running.  The  subsystem  would  statistically  analyze  responses.  Please  stale 
your  opinion  on  this  approach  to  evaluating  system  performance.  Does  the  subject  system  contain 
such  a  built-in  evaluator  (or  any  other  kind  of  evaluator)? 

SV5.  What  kind  of  tools  (built-in  or  otherwise)  and  techniques  did  you  use  to  evaluate/ validate 
the  A I  system? 


SV6.  When  testing  the  subject  system,  what  criterion  did  you  use  to  judge  whether  the  system 
passed  or  failed  a  test?  Please  elaborate  for  ail  levels  of  test  (e  g.  for  testing  a  specific  rule  and  for 
testing  the  system  as  a  whole).  What  criterion  was  used  to  judge  that  the  system  was  ready  for 
use? 


SV7.  In  rapid  prototyping,  the  general  approach  is  “build  a  little,  test  a  little”  If  rapid  prototyping 
was  used  for  the  subject  system,  please  describe  the  method  of  system  evaluation.  Did  the  testing 
phase  become  more  rigorous  as  the  system  matured? 


FIELD  SUPPORT 


The  following  questions  apply  only  to  those  systems  that  are  actually  being  used  in  the  field.  If  use 
of  the  subject  system  is  limited  to  a  research  and  development  environment,  proceed  to  the  next 
section. 


FSl.  Was  performance  an  issue,  either  in  terms  of  response  time  or  degree  of  accuracy?  If  applica¬ 
ble,  describe  the  effort  taken  to  improve  system  performance  and  its  effect  on  phases  of  the  project 
that  had  been  frozen. 


FS‘2.  Were  any  enhancements  suggested  by  the  users  as  they  learned  to  work  with  the  system?  If 
applicable,  describe  the  mechanism  for  implementing  the  suggested  changes. 


1  Cercone,  N.,  et  al,  “Designing  and  Automating  the  Quality  Assessment  of  a  Knowledge- Based  System:  The  Initial 
Automated  Academic  Advisor  Experience",  IEEE  1984  Workshop  On  Principles  Of  Knowledge- Based  Systems, 
IEEE  Catalog  Number  84CH2104-8. 
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ATTACHMENT  1 


GENERAL  DEFINITION  OF  TERMS 

Language  Processor.  A  language  processor  mediates  information  exchanges  between  the  expert 
system  and  the  user  by  processing  questions,  commands  and  volunteered  information  expressed  in 
a  problem-oriented  language. 

Knowledge  Base.  A  knowledge  base  is  a  repository  for  rules,  facts,  and/or  information  about 
the  problem  to  be  solved. 

•  Rules  -  This  part  of  the  knowledge  base  contains  procedural  interpretations. 

•  Facts  -  This  part  of  the  data  base  contains  declarative  (i.e.,  non-procedural)  information 
pertinent  to  the  expert  system  domain. 

Justifier.  A  justifier  explains  to  the  user  why  certain  conclusions  were  reached  and  why  others 
were  not. 

Interpreter.  An  interpreter  executes  the  chosen  agenda  item  by  applying  the  corresponding 
knowledge  base  rule. 

Scheduler.  A  scheduler,  which  may  contain  a  fair  amount  of  knowledge  in  its  own  right,  controls 
the  agenda  by  determining  which  pending  action  should  be  executed  next. 

Consistency  Enforcer.  A  consistency  enforcer  attempts  to  maintain  a  uniform  representation 
of  the  evolving  solution  by  applying  some  quantitative  scheme  to  determine  the  degree  of  belief  in 
each  decision. 

Communications  Handler.  A  communications  handler  manages  the  interfaces  between  compo¬ 
nents  in  a  hybrid  or  multi-component  system  (e  g.  blackboard). 
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Appendix  C 
Case  Summaries 


C.l  ARINC  Summary 

The  ARINC  System  Testablility  and  Maintenance  Program  (STAMP),  allows  an  engineer  to  analyze 
a  system’s  testability  for  field  maintenance  operations.  The  system  will  recommend  design  changes, 
and  provide  fault  isolation  strategies  for  manual,  semi-automatic  or  automatic  field  fault  isolation. 

The  development  team  was  comprised  of  four  to  five  engineers,  with  no  more  than  three  computer 
programmers  assigned  at  any  one  time.  Two  of  the  engineers  had  previous  Ai  experience.  Three 
of  the  project  personnel  had  DOD  software  development  experience.  Nonetheless,  only  internal 
standards  were  used  during  the  prototype  and  development  phases. 

STAMP  was  funded  as  an  IR&D  project.  The  initial  prototype  was  released  within  6  months 
followed  by  a  field  model  which  was  developed  within  two  years.  The  system’s  success  is  based  on 
the  fact  that  STAMP  demonstrated  the  possiblity  of  a  new  technology  and  significantly  increased 
productivity  by  automating  a  manual  process.  STAMP  is  currently  an  established  product  at 
ARINC. 

There  was  no  requirements  analysis  for  the  prototype.  However,  a  requirements  analysis  was  per¬ 
formed  for  the  rehosted  version  of  STAMP  from  an  Apple  to  the  HP-1000  operating  system.  The 
only  formal  software  development  procedures  followed  for  the  initial  prototype  were  during  prob¬ 
lem  formulation  which  encompassed  some  of  the  ideas  behind  identification  (determining  problem 
characteristics),  conceptualization  (finding  concepts  to  represent  knowledge)  and  formalization  (de¬ 
signing  structures  to  organize  knowledge,  i.e.,  tool  building).  Throughout  the  prototype  phase,  the 
capability  of  the  system  was  continually  assessed.  During  development  of  the  rehosted  version 
of  STAMP,  more  rigid  standards  were  followed.  More  attention  was  placed  on  finding  ways  to 
represent  knowledge  and  formalisms  were  developed.  Extensive  documentation  was  required. 

The  knowledge  acquisition  is  inherent  to  STAMP.  A  new  knowledge  base  is  implemented  by  a 
testing  expert  with  each  new  use  of  the  tool. 

STAMP  is  comprised  of  the  following  components:  a  knowledge  base,  an  interpreter,  a  consis¬ 
tency  enforcer  and  a  compiler.  Tools  used  during  development  encompassed  a  Fortran  77  com¬ 
piler/debugger  and  the  HP-1000  operating  system. 

The  STAMP  system  is  written  as  conventional  software  in  Fortran  77.  The  system  feasibility  was 
tested  by  individuals  who  checked  code  at  the  module  and  integration  level  Users  were  allowed  a 
two  month  trial  period  to  shake  out  report  problems  with  the  system  A  user  conference  was  then 
held  to  determine  final  action  items. 

Overall,  the  system  underwent  five  major  architectural  changes  which  allowed  for  incremental 
testing  and  iterative  system  development.  Some  of  the  lessons  learned  were  to  develop  a  simple, 
usable  prototype  excluding  bells  and  whistles;  document  early  and  frequently;  free  the  experis  from 
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administrative  burdens  to  allow  them  the  time  and  opportunity  to  develop  ideas;  and  encourage 
creativity  during  the  prototyping  stage. 

C.2  Boeing  Computer  Services  Summary 

Boeing  Computer  Services  reported  on  a  Strategic  Force  Management  Decision  Aid  which  is  a 
know  ledge- based  replanner.  The  inputs  to  the  system  are:  a  description  of  a  previously  created 
plan  for  the  employment  of  strategic  forces,  and  a  description  of  an  event  whirh  would  require 
alteration  to  the  plan.  The  system  then  determines  a  suitable  modification  to  the  plan  which  is 
presented  to  the  user  for  review  and  approval. 

The  project  was  developed  with  internal  funds  by  one  engineer  with  previous  AI  experience.  The 
first  prototype  was  developed  in  six  man-months,  and  the  system  is  not  currently  fielded.  Boeing 
Computer  Services  had  previously  built  over  30  prototype  AI  systems.  In  terms  of  demonstrating 
the  feasibility  of  a  new  technology,  the  system  is  a  success.  Overall  user  satisfaction  and  significant 
productivity  gains  were  also  substantial. 

The  model  used  to  develop  the  Strategic  Force  Management  Decision  Aid  consisted  of  the  following 
steps: 

•  ELICITATION  -  Knowledge  is  acquired  from  information  sources  to  produce  an  information 
base. 

•  ANALYSIS  -  The  information  base  is  analyzed  and  structured  to  produce  a  knowledge  base. 

•  TESTING  -  The  knowledge  base  content  is  tested  to  produce  a  case  base. 

•  REFINING  -  The  case  base  is  refined  to  produce  an  expert  knowledge  base. 

•  COMBINING  -  Multiple  expert  knowledge  bases  in  the  same  or  related  problem  domains  are 
combined  to  form  a  knowledge  network. 

•  TESTING  -  The  contents  of  the  knowledge  network  is  tested  to  produce  a  knowledge  network 
case  base. 

•  TAILORING  -  The  knowledge  network  case  base  is  tailored  to  the  requirements  of  the  specific 
customer  to  produce  a  delivered  knowledge  based  system. 

•  RECORDING  -  The  delivered  knowledge  based  system  may  incorporate  recording  of  infor¬ 
mation  that  is  used  as  an  information  source  for  further  elicitation. 

These  steps  were  iterated  several  times,  with  feedback  from  subsequent  steps  used  to  revise  the 
results  of  previous  steps.  Three  review  sessions  with  the  ‘customer’  were  held  which  included 
demonslraf  ions  of  the  current  level  of  system  functionality  and  an  exchange  of  information,  ideas 
and  concepts. 

A  requirements  analysis  phase  was  performed  which  consisted  of  discussion  sessions  with  the  'cus¬ 
tomer'  to  identify  types  of  problems  requiring  a  solution  which  uses  the  replan  approach.  For  each 
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problem  identified,  a  suitable  problem  solving  methodology  was  determined.  The  set  of  problems 
and  their  corresponding  solution  methodologies  were  used  as  a  definition  of  requirements.  User 
input  to  this  process  was  considered  very  useful. 

Four  experts  were  involved  in  the  process  of  knowledge  acquisition  which  consisted  of  informal 
interviews.  A  negotiation  process  was  used  to  resolve  any  differences.  The  experts  were  chosen 
based  on  their  years  of  experience  and  the  community  acknowledgement  of  the  person’s  status 
as  an  expert.  The  Knowledge  Engineer  was  also  an  expert  in  the  field  and  provided  additional 
verification  of  the  experts  accuracy.  A  frame  based,  logic  based  and  production  rule  representation 
was  used  to  encode  the  knowledge.  Ill-defined  and  incomplete  knowledge  is  handled  through  the 
implementation  of  alternate  paths  through  the  production  rules  in  a  manner  similar  to  default  logic. 

The  tool  KEE,  operating  on  a  Texas  Instruments  Explorer  Lisp  Machine  was  used  to  develop  the 
system.  A  rapid  prototyping  approach  was  also  used  in  the  development  process.  The  evolving 
prototype  was  subject  to  review  on  a  continuous  basis  informally,  and  formally  at  about  two-month 
intervals.  There  was  no  approval  necessary  for  design  changes,  and  no  controls  were  used  to  track 
the  updated  versions  of  the  system. 

The  components  of  the  system  include: 

•  Language  Processor 

•  Knowledge  Base 

•  Justifier 

•  Interpreter 

•  Consistency  Enforcer 

In  terms  of  size,  the  knowledge  base  consisted  of  50  productions  rules,  approximately  45  frames  used 
as  templates,  and  8000  lines  of  LISP  code.  During  execution  of  a  typical  scenario,  the  knowledge 
base  grows  to  300  or  more  frames  and  several  hundred  logic  based  facts. 

A  set  of  test  scenarios  was  used  to  validate  the  system.  No  formal  test  plans  were  developed.  The 
success  criterion  used  was  to  examine  the  plan  generated  by  the  system  and  determine  if  it  would 
be  considered  an  acceptable  plan  by  an  expert. 


C.3  Boeing  Military  Airplane  Company  Summary 

The  system  reported  on  by  the  Boeing  Military  Airplane  Company  in  Wichita,  Kansas,  is  an 
Internal  Research  ami  Development  (IR&D)  project  called  .1/ for  Automatic  Tnrgtf  Recognition 
(ATIf).  The  purpose  of  the  system  is  to  reduce  problems  in  the  modern  battlefield  environment 
l>V  improving  present  target  recognizers  through  A I  techniques.  This  long  term  research  project  is 
currently  in  the  feasibility  demonstration  phase 

The  Boeing  ATI!  software  development  team  included  one  lead  engineer  with  approximately  twelve 
years  of  conventional  software  research,  development  and  additional  personnel  knowledgeable  in 


m 


C  c.'C 
<v\-v 

•.y.yjv, 

vy.-v 

*m  V  % 

A  V  V  V 


l 


V  %  %  1 

*  s  .*  V* . 
v  %  \ 

*  «■*  V  V 


r.’.vv, 

v'-V'vS 

v  v  v  • 

VV>. 


*  v\"i 
/•  /•  A 

y.;-.- 

CvS'I** 


5 


»  i  *  • 

■yv.v 


cv'C-vv-’d 


C.3  Boeing  Military  Airplane  Company  Summary 


various  areas  including  image  processing.  The  lead  engineer  had  also  completed  an  in-houae  AI 
Associate  Training  course.  All  involved  personnel  had  considerable  experience  with  DOD  and  com¬ 
mercial  software  development.  Standards  for  conventional  software  development  had  a  significant 
influence  on  the  AI  software 

The  general  process  used  in  the  development  of  ATR  to  date  consisted  of  the  following  phases. 


•  Identification  -  a  study  was  undertaken  to  determine  military  product  arena  that  were  suitable 
for  improvement  via  AI  technology  (i.e.,  ATR/image  understanding) 

•  Conceptualization  -  conceptualization  of  problem  characteristics  derived  throughout  the  Iden 
tification  phase  into  system  capabilities  (i.e.,  dealing  with  sensor  fusion  capability,  incomplete 
or  uncertain  data) 

•  Formalization  -  considered  a  particularly  valuable  phase  in  that  the  expert  system  development 
tool  was  selected  which  determined  the  form  of  knowledge  representation  for  the  ATR  system 

•  Implementation  -  gathering  knowledge  via  literature  surveys  with  minimal  expert  consultation 

•  Testing  -  validating  the  knowledge  base  and  system  behavior  (i.e.,  demonstrating  system 
concepts) 
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Each  of  the  above  phases  is  considered  appropriate  for  the  development  of  an  AI  system.  How¬ 
ever,  it  was  perceived  that  as  additional  information  regarding  the  problem  was  encountered,  an 
iterative  phase  which  produced  a  new  prototype  model  for  each  reiteration  was  necessary  to  the 
development  of  the  ATR  software.  The  cyclic  phase  deviates  from  DOD  and  company  standards 
in  that  requirements  are  not  fixed. 

Knowledge  acquisition,  which  is  never  considered  complete,  consisted  of  a  literature  survey  in  the 
areas  of  image  understanding.  Documentation  was  prepared  with  known  techniques,  and  algorithms 
for  image  understanding  were  reviewed  by  internal  company  personnel. 

The  knowledge  representation  method  of  frames  and  rules  was  determined  by  the  type  of  data 
and  available  tools.  The  knowledge  base  continues  to  expand  with  informal  reviews  and  a  formal 
approval  process.  Tools  used  in  developing  the  software  were  Knowledge  Craft  (in  particular  OPS-5) 
and  LISP. 

The  major  components  of  the  system:  a  knowledge  base,  justifier  and  an  inference  engine,  were 
largely  determined  by  cost  and  schedule  constraints.  In  addition,  the  architectural  components 
were  influenced  by  the  tools  available  and  were  subject  to  review  and  approval. 

The  ATR  system  has  not  reached  a  stage  in  which  extensive  project  performance  can  be  evaluated. 
As  the  system  reaches  completion,  and  is  ready  for  deployment,  user  satisfaction  will  be  examined 
as  well  as  productivity.  The  feasibility  of  the  system  is  determined  by  the  ability  to  demonstrate 
concepts  as  in  the  original  proposal  and  the  evolving  concept  definition.  Every  rule  was  triggered 
and  fired.  However,  not  every  combination  was  tested.  The  system  passed  testing  when  a  few 
image  inputs  produced  the  appropriate  output  result.  In  addition,  the  user  interface  is  continually 
tested.  As  each  prototype  was  released,  the  main  testing  target  concentrated  on  those  areas  where 
changes  had  been  made. 
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C.4  Brattle  Research  Corporation  Summary 


Overall,  the  ATR  system,  while  not  an  expert  system  in  the  classic  sense,  has  a  domain  which 
is  considered  much  more  complex  than  the  typical  expert  system.  As  a  research  project,  the 
reliance  on  literature  that  influences  and  causes  continual  changes  to  the  requirements,  rest  ills  m  an 
iterative  process  through  all  the  development  phases  and  ends  in  a  new  prototype  The  flexibility 
of  a  research  area  such  as  the  ATR  would  suffer  without  the  ability  to  rework  requirements  to 
accomodate  important  data  or  changes  to  information. 


C.4  Brattle  Research  Corporation  Summary 


Using  both  venture  and  contract  funding,  Brattle  Research  Corporation  is  building  a  generalized 
text  extraction  system  focusing  on  business  information.  The  sources  of  business  information  are 
online  database  services  such  as  Dow  Jones  News/ Retrieval  and  wire  services  such  as  Businesswire 
and  PR-Newswire.  The  system  has  two  basic  functions:  topic  recognition/retrieval  and  extraction 
The  topic  recognition  side  of  the  system  has  gone  through  a  feasibility  study,  several  prototypes  and 
is  being  appled  in  staff  study  contracts.  In  terms  of  breadth  and  depth  of  retrieval,  the  intelligent 
topic  recognition  aspect  has  been  found  to  be  much  more  accurate  than  currently  available  keyword 
search  techniques. 

Several  extraction  applications  have  been  tested,  and  a  general  extraction  utility  is  now  in  tin- 
engineering  design  phase.  User  interface  issues  are  still  in  a  very  early  stage. 

The  development  environment  for  the  system  is  based  on  the  Symbolics  3600  using  Lisp.  For 
portability  reasons,  Brattle  Research  is  moving  from  Zeta-Lisp  to  Common  Lisp.  The  company 
has  built  its  own  database  management  system,  information  retrieval  language,  and  their  own  text 
analysis  tools.  The  tools  were  a  tremendous  aid  in  facilitating  rapid  prototyping  and  conceptual 
bread  boarding. 

Brattle  Research  employs  a  relatively  organized  development  strategy:  define  overall  project  struc¬ 
ture,  identify  milestones  and  allocate  budget  vs.  phase.  Documentation  includes  conceptual  system 
specifications  and  design  documents  which  contain  timetables  and  budgets.  All  documentation  re¬ 
ceives  both  managerial  and  technical  review  with  feedback  in  terms  of  consensus  rather  than  strict 
approval/disapproval.  Weak  areas  identified  by  the  review  process  were  focal  points  during  the 
prototyping  phase. 

The  project  team  currently  consists  of  seven  people  all  with  more  than  seven  years  experience  in 
the  A I  field.  The  overall  projected  team  size  is  estimated  at  twelve. 

For  each  topic,  the  knowledge  acquisition  process  is  based  on  working  with  someone  w  ho  is  familiar 
with  the  way  a  topic  is  described  in  the  literature.  The  knowledge  engineer  works  to  deduce 
a  set  of  patterns  that  describes  the  topic.  The  knowledge  base  is  then  comprised  of  linguistic 
patterns  associated  with  particular  topics.  The  inference  mechanism  includes  a  variety  of  techniques 
collectively  described  as  a  simple  form  of  discourse  analysis.  Specific  methods  include  pattern 
recognition  (including  some  signal  processing  strategies  -  i.e.  treat  text  as  a  stream  of  symbols), 
and  linguistic  mechanisms  such  as  chart  parsing,  categorization  of  syntactic  elements  and  treatment 
of  context-free  grammars. 

In  terms  of  user  involvement,  existing  customers  have  been  providing  feedback  on  the  prototypes 
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SI 


Before  releasing  its  product.  Brattle  Research  plans  to  form  a  scientific  advisory  hoard  to  comment 
on  the  system 

Brattle  uses  Symbolics'  configuration  management  system  augmented  with  its  own  code  to  control 
software  distribution.  One  of  the  Symbolics  machines  is  a  dedicated  file  server  which  collects  all 
incoming  information  in  a  central  database  repository. 

Test  data,  which  consumes  70  megabytes  of  memory,  is  based  on  articles  from  The  Wall  Street 
Journal,  Electronics  magazine,  PR-Newswire,  etc.  To  date,  the  testing  criteria  has  been  absolute 
accuracy.  Both  regression  analysis  and  blind  tests,  i.e.  -  test  sets  that  haven’t  been  processed 
before  -  have  been  used 

The  system  contains  code  to  automatically  compile  the  abstract  descriptions  of  grammars  and 
documents  into  machine  code  for  quickly  searching  text. 


C.5  Carnegie  Group  Inc.  Summary 


The  Carnegie  Group  Inc.  reported  on  the  DISPATCHER  System  which  monitors  and  controls  a 
factory  floor  Automated  Materials  Handling  System.  The  system,  a  contractual  project,  took  thirty- 
six  man-months  to  deliver.  The  prototype  release  took  twenty-one  man-months.  The  system’s 
success  is  based  upon  the  product’s  feasibility,  user  satisfaction  and  productivity  gains. 

The  development  team  consisted  of  three  engineers  of  which  two  had  prior  AI  software  development 
experience.  None  of  the  engineers  had  any  exposure  to  DOD  software  development  standards. 

The  general  development  process  for  the  DISPATCHER  system  consisted  of  the  following  standard 
phases: 

•  Identification  -  determining  problem  characteristics; 

•  Conceptualization  •  finding  concepts  to  represent  knowledge; 

•  Formalization  -  designing  structures  to  organize  knowledge  (tool  building); 

•  Implementation  -  encoding  knowledge;  and 

•  Testing  -  validating  a  knowledge  base  and  system  beiiavior. 

However,  the  development  process  did  deviate  from  the  above  methodology  in  two  ways.  First, 
knowledge  acquisition  was  an  additional  phase  that  began  after  identification  and  before  concep¬ 
tualization  Secondly,  the  DISPATCHER  System  was  developed  incrementally  which  involved 
stepping  through  each  phase  iteratively  until  the  system  was  sufficiently  refined.  The  product  was 
continuously  updated  up  to  and  after  the  final  installation 

Although  it  appears  that  a  requirements  analysis  was  not  performed,  a  specification  document  was 
produced  The  development  effort  was  bounded  by  the  document  and  by  consultation  with  the 
purchaser.  After  installation  the  development  was  bounded  by  negotiation  between  the  vendor  and 
purchaser 


C.6  Digital  Equipment  Corporation  Summary 

Knowledge  acquisition  consisted  of  information  obtained  from  the  specification  documentation  and 
from  informal  contact  with  the  intended  users.  No  domain  experts  were  available  for  consultation 
on  the  domain. 

The  DISPATCHER  System  is  comprised  of  the  following  components:  a  separate  ride  and  fai  l 
knowledge  base,  an  inference  engine  and  a  communications  handler.  Tools  used  during  development 
included  the  OPS5  and  a  code-generator  for  external  database  routine.  The  use  of  OPS5  mandated 
separate  fact  and  rule  knowledge  bases  while  domain  considerations  required  a  communications 

handler. 

The  DISPATCHER  System’s  functionality  was  tested  by  the  user.  A  simulation  tool  was  built 
as  a  means  by  which  the  major  system  with  which  DISPATCHER  communicates  could  be  tested 
Testing  was  considered  complete  on  the  basis  of  casual  and  random  tests  of  the  system  functionality, 

Overall,  response  time  and  accuracy  were  main  concerns  throughout  the  development  effort  Users 
provided  continuous  functionality  improvement  suggestions  to  the  engineers  who  attempted  to 
incorporate  their  ideas.  A  recommendation  to  commit  the  user  to  a  more  detailed  specification  of 
the  system  was  made.  Also,  the  system  architecture  could  have  been  more  carefully  defined  and 
reviewed  at  an  earlier  stage  in  the  system  development. 

C.6  Digital  Equipment  Corporation  Summary 

Digital  Equipment  Corporation  (DEC)  reported  on  XCON,  an  expert  system  that  configures  DEC 
computer  systems.  Specifically,  XCON  accepts  a  list  of  items  from  a  customer  order,  configures 
them  into  a  system,  notes  additions/deletions  made, and  prints  out  a  set  of  detailed  diagrams 
depicting  the  spatial  relationships  amongst  the  components.  With  the  automation  of  a  formerly 
manual  task,  XCON  has  decreased  the  number  of  costly  configuration  errors  and  significantly 
increased  customer  order  processing  speed. 

The  development  of  XCON  began  in  late  1978  at  Carnegie-Mellon  University  (CMU)  where  it  was 
known  as  the  Rl  system.  In  early  1980,  it  was  installed  at  the  first  DEC  plant  and  used  on  a  daily 
basis.  By  January  1981,  DEC  no  longer  required  assistance  from  CMU  in  terms  of  maintaining 
and  developing  XCON.  Since  that  time,  XCON  has  become  a  mature  system  and  is  currently  in 
the  “production  mode”  phase. 

It  is  difficult  to  discuss  XCON  without  mention  of  the  adjunct  expert  system  XSEL  (expert  -elling 
tool)  By  submitting  orders  to  XCON,  XSEL  helps  the  DEC  sales  personnel  interactively  configure 
computer  systems  to  prepare  accurate  quotes  and  match  specific  products  to  customer  needs  XSEL 
and  XCON  share  the  same  knowledge  base  and  together  contain  more  than  H).(KK)  rules  Because 
the  knowledge  base  is  so  very  dynamic,  (i.e.  new  components  are  frequently  introduced  existing 
components  are  often  modified),  an  upgraded  version  of  XSEL  XCON  is  released  quarterly.  Over 
(lie  past  lew  years,  DEC  has  adopted  a  formal  release  procedure  which  consists  of  the  following 
lour  phases 

•  I’lanuing; 

•  Development; 
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System  Quality  Review  (SQR);  and 
Release 


The  procedure  is  iterative  from  Planning  through  SQR.  In  the  Planning  phase,  the  workload  is 
prioritized,  checks  are  made  in  the  OPS5  code  for  dependency  levels  and  interim  target  dates 
established  as  a  function  of  the  scheduled  release  date 

In  the  development  phase,  knowledge  acquisition  occupies  a  large  portion  of  the  effort.  Multiple 
experts  are  involved  for  each  new  product.  Technical  design  reviews  are  held  at  the  team  level  and 
implementation  of  the  changes  within  the  nine  VAX  cluster  environment  occurs.  Correctness  is 
verified  using  several  internally  developed  tools  which  perform  checks  on  such  things  as  rule  syntax 
and  database  entries  The  VMS  configuration  management  system  (CMS)  is  used  to  control  all 
source  files  Testing  on  the  changes  made  (similar  to  unit  testing)  is  also  performed  during  this 
phase. 

In  the  SQR  phase,  a  project  test  plan  is  established  and  executed  against  a  large  set  (>  1000)  of 
hypothesized  customer  orders.  The  testing  criteria  is  wholly  qualitative,  namely,  ir  there  are  no 
major  problems  that  are  foreseen  to  adversely  impact  th<  business,  the  new  version  of  the  system 
is  approved  Following  a  successful  SQR,  the  Release  phase  begins.  The  target  DEC  facilities 
receive  a  new  tape  of  XSEL/XCON,  an  installation  checklist,  an  installation  systems  management 
guide,  an  on-line  summary  of  new  parts  and  system  functionality  and,  if  necessary,  a  summary  of 
significant  problems  yet  to  be  resolved. 

The  project  team  currently  consists  of  35  staff  divided  into  the  following  groups: 

•  Administration  (3  people); 

•  lTser  Support  (5  people); 

•  Technical  Support  -  (6  people);  and 

•  Knowledge  Engineering  (21  people). 

'l’lie  Technical  Support  team  deals  with  the  non-OPS5  code  -  there  are  5  conventional  languages 
that  comprise  the  system  as  well  as  many  databases.  The  Knowledge  Engineering  staff  is  typically 
sub-divided  into  2-3  teams  whose  responsibility  is  knowledge  acquisition  and  representation,  OPS5 
coding,  and  testing.  The  project  is  run  using  a  management  team  concept  wherein  the  project 
manager  makes  very  few  decisions  without  consulting  pertinent  team  members. 


C.7  Expert  Technologies,  Inc.  Summary 

Expert  Technologies,  Inc.  (ETI)  reported  on  their  PEGASYS  system  which  is  an  expert  system 
for  the  automatic  pagination  of  yellow  page  directories.  The  system  has  three  modes  of  operation: 
batch,  review,  and  development,  In  the  batch  mode,  PEGASYS  provides  automatic  pagination 
using  heuristics  In  the  review  mode,  the  user  can  review  the  system’s  pagination  for  quality 
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C.  7  Expert  Technologies ,  Jnc.  Simunary 


control  and  interactively  make  corrections.  The  corrections  can  be  made  manually  -  graphics 
object  composition,  or  they  can  be  inputted  into  the  knowledge  base  -  changing/roRning  a  rule  to 
yield  better  pages.  In  the  development  inode,  the  user  has  the  ability  to  design  new  rule  bases  and 
examine  the  resulting  new  products. 

PEGASYS  was  an  internally  funded  project  and  ETI’s  first  A1  system.  The  development  team 
consisted  of  7  to  8  engineers,  all  with  LISP  experience.  Three  members  of  the  team  also  had  prior 
AI  experience. 

Seven  of  the  team  members  developed  the  first  two  prototypes.  The  first  one  was  developed  in  one 
month  and  the  second  in  two  months.  Eight  team  members  developed  the  third  prototype  and  the 
deliverable  system.  Each  took  three  months  to  develop.  The  PEGASYS  system  was  delivered  in 
69  man-months,  three  months  ahead  of  schedule. 

PEGASYS  is  considered  a  success  in  terms  of  demonstrating  the  feasibility  of  a  new  technology  and 
overall  user  satisfaction.  The  system’s  most  significant  success  lies  in  its  flexibility  which  results  in 
a  system  that  can  be  extended  and  maintained  easily. 

A  requirements  analysis  phase  was  performed  which  included  user  input.  The  tools  KEE  and 
Knowledge  Craft,  along  with  Xerox  machines  and  Tl  Explorers  were  utilized  in  this  phase.  A 
functional  specification  was  generated  which  described  90%  of  the  eventual  system's  functionality 
The  requirements  were  frozen  at  the  end  of  this  phase. 

Knowledge  acquisition  was  achieved  through  verbal  communication  between  the  domain  experts 
and  the  knowledge  engineers.  Extensive  documentation  and  prototyping  was  performed  during 
this  process.  The  documentation  was  reviewed  by  the  senior  design  engineer  for  approval,  and 
the  prototypes  were  integrated  into  the  system  upon  approval.  The  choice  of  experts  was  obvious 
as  there  are  only  a  few  people  who  have  special  pagination  expertise.  Three  experts  were  used 
and  knowledge  from  either  one  was  encoded  and  results  approved  by  the  other  two.  This  process 
allowed  for  complete  conflict  resolution. 

The  knowledge  representation  method  consists  of  a  semantic  network  of  frames.  This  method  was 
chosen  because  of  the  close  match  it  provided  with  the  domain  knowledge.  Incomplete  know  ledge 
was  encoded  procedurally.  The  knowledge  base  expanded  during  the  development  process  with 
approval  by  2  senior  engineers.  PEGASYS  allows  for  extension  of  its  knowledge  base  on  an  as- 
required  basis. 

There  were  three  major  break  points  during  the  development  process  where  new  designs  were 
encoded.  This  resulted  in  a  complete  reimplementation  of  the  system  at  the  end  of  the  second 
break  point.  Approval  for  these  changes  came  from  the  project  manager  and  two  senior  know  ledge 
engineers  An  in-house  proprietary  configuration  management  approach  was  used  to  track  updated 
versions  of  the  system  during  development. 

The  PEGASYS  system  has  the  following  characteristics: 

•  A  simple  inference  engine 

•  Knowledge  encoded  in  terms  of  semantic  primitives 

•  Meta-rules  encoded  as  “process”  rules  (procedures  and  frames/semantic  objects) 
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•  Consistency  Enforcer 

The  architecture  was  reviewed  and  approved  by  top  A!  specialists  in  industry  and  academia.  The 
user  interface  was  done  conventionally,  and  was  one  of  the  two  largest  parts  of  the  system. 

Development  tools  were  used  in  building  the  prototypes.  The  tools  contributed  to  development 
frustrations  because  of  the  expertise  of  the  AI  programmers.  Consequently,  development  was 
moved  entirely  to  LISP  programming. 

An  acceptance  test  plan  (ATP)  for  the  system  was  written  before  the  start  of  the  project.  This 
plan  detailed  acceptability  with  respect  to: 

•  Productivity 

•  Efficiency 

•  Software  maintenance  and  performance 

System  performance  was  evaluated  relative  to  the  current  in-house  systems  (i.e.  manual  and  semi¬ 
automatic  pagination).  In  terms  of  field  support,  fine  tunings  of  the  system  architecture  were 
incorporated  to  enhance  the  performance  of  PEGASYS  in  terms  of  response  time  and  degree  of 

accuracy. 


C.8  Frey  Associates,  Inc.  Summary 

Frey  Associates,  Inc.  reported  on  the  Themis  TM  Management  Information  System.  Themis  is  a 
natural  language  processing  system  that  answers  English  requests  about  information  stored  in  a 

database. 

1  he  project  team  started  with  6  staff  members  and  eventually  grew  to  12  members.  One  person 
had  previous  AI  experience.  Since  most  of  Frey’s  projects  are  system  oriented,  as  opposed  to  Data 
Processing  (DP)  efforts,  it  is  surmised  that  most  of  the  software  engineers  had  DOD  conventional 
software  development  experience. 

Themis  was  funded  internally  and  followed  a  schedule  defined  by  management.  The  first  prototype 
was  completed  within  6  months.  The  product  before  beta  fielding  took  approximately  6  man  years 
with  the  final  product  encompassing  about  10  man  years. 

I  he  general  process  used  in  the  development  of  Themis  consisted  of  the  following  phases: 

•  Identification  -  determining  problem  characteristics 

•  Conceptualisation  -  finding  concepts  to  represent  knowledge 

•  Formalization  -  designing  structures  to  organize  knowledge  (tool  building) 

•  Implementation  -  capturing  knowledge 

•  I'esting  -  validating  a  knowledge  base  and  system  behavior 


C.9  GTE  Data  Service  a  Summary 


Documentation  was  provided  at  the  end  of  the  Identification,  Conceptualization  and  Formalization 
phases.  The  Implementation  phase  was  divided  into  2  stages,  prototype  and  actual  product.  An 
iteration  from  Identification  to  Implementation  occured  at  least  once.  From  then  on,  iterations  were 
primarily  between  Formalization.  Implementation  and  Testing  (4  +  ).  Testing  was  used  to  provide 
a  basis  for  the  user  documentation  of  the  system. 

As  a  natural  language  system,  Themis  is  different  from  an  expert  system  in  the  approach  used  for 
knowledge  acquisition.  The  goals  of  the  system  were  limited  to  the  inquiry  characteristics  of  one 
data  base  from  which  both  the  requirements  and  user  interface  plans  were  derived.  The  user's  main 
contribution  to  the  system  effort  was  in  defining  the  user  interface  and  the  types  of  utilities  needed 
to  support  the  types  of  queries  needed.  The  user  was  also  valuable  in  defining  types  of  queries  for 
Themis. 

The  knowledge  representation  method  used  was  rule-based  since  it  was  regarded  by  the  design 
team  as  most  appropriate  to  natural  language  processing.  The  main  programming  tool  used  was 
InterLISP  which  included  the  Programming  Assistant,  DWIM,  Masterscope,  a  structure  editor, 
and  debugging  tools.  Source  management  tools  were  also  used. 

During  the  development  process,  a  series  of  prototypes  were  generated.  Major  design  changes  were 
approved  and  implemented  throughout. 

Themis  is  comprised  of  the  following  components:  Language  Processor,  Rule-Based  Knowledge 
Base,  Justifier,  Interpreter,  Scheduler,  Consistency  Enforcer,  and  a  Communications  Handler.  The 
AI  software  was  interfaced  through  mail  boxes  to  the  conventional  software. 

Themis  is  self  modifiable  in  that  when  vocabulary  and  other  new  rules  are  introduced,  Themis  will 
evaluate  contradictions  and  generate  new  rules  to  clarify  these  contradictions. 

Test  and  regression  procedures  are  in  place  for  Themis  which  include  correct  as  well  as  incorrect 
queries.  The  testing  data  base  is  continually  enhanced  by  internal  testers,  customers  and  users. 
Due  to  the  self  modifying  nature  of  Themis,  tests  are  repeated  without  ending  sessions  to  ensure 
consistent  results.  User  involvement  is  considered  important  during  this  phase  and  throughout  the 
lifetime  of  the  system.  Themis  has  built  in  testing  capabilities  along  with  automated  error  logging 
facilities. 

Overall,  the  development  process  used  by  Frey  Associates,  Inc.  has  proven  quite  successful.  If  any 
phase  were  to  be  enhanced,  it  would  be  the  testing  phase  to  ensure  completeness  of  the  know  ledge. 

C.9  GTE  Data  Services  Summary 

GTE  Data  Services  reported  on  their  Central  Office  Printout  Analysis  and  Suggest  System  (COM¬ 
PASS)  project.  COMPASS  is  an  expert  system  designed  to  diagnosis  problems  with  GTE's  no  2 
EAX  switch.  As  input,  COMPASS  receives  a  data  file  which  contains  error  messages  produced  by 
a  particular  switch.  The  output  COMPASS  generates  consists  of  problem  and  fault  identification 
and  maintenance  suggestions. 

This  was  GTE  Data  Services  first  experience  with  an  AI  project.  The  project  team  consisted  of  two 
to  seven  people  depending  on  the  project  phase.  One  team  member  had  previous  A I  experience 
and  another  had  limited  AI  experience. 
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C.9  GTE  Data  Services  Summary 


COMPASS  was  internally  funded  aa  a  feasibility  study.  Management  defined  a  schedule  where  the 
prototype  would  be  built  in  36  months  and  three  months  later  the  system  would  be  in  a  limited 
field  study. 

The  success  criteria  for  COMPASS  are:  technology  feasibility,  user  satisfaction,  and  productivity 
gains.  COMPASS  has  been  rated  as  a  success  in  technology  feasibility,  as  it  has  demonstrated  the 
usefulness  of  expert  systems  for  some  telecommunications  problems.  Currently  the  system  is  in  a 
limited  field  study  and  the  users  appear  to  be  satisfied  with  the  content  and  form  of  COMPASS 
output.  Success  in  terms  of  productivity  gains  has  not  yet  been  determined. 

The  stages  of  development  in  the  COMPASS  project  are  as  follows: 

•  Identification  -  this  phase  was  emphasized  due  to  the  system’s  origin  as  a  feasibility  study. 
An  internal  technical  note  and  a  paper  were  produced  in  this  phase. 

•  Conceptualization  -  In  this  phase  several  internal  technical  notes  were  produced. 

•  Formalization  -  A  software  control  knowledge  base  was  added  to  the  system  to  track  the 
various  knowledge  bases  and  method  files. 

•  Implementation  -  Knowledge  acquired  from  the  expert  was  collected  into  several  documents 
referred  to  as  Knowledge  Acquisition  Rules  (KAR)  documents 

The  above  phases  were  iteratively  visited,  especially  early  in  the  project.  During  these  phases, 
internal  reviews  were  held  to  report  progress  and  to  establish  conventions  among  the  various  de¬ 
velopers.  A  requirements  analysis  phase  was  not  performed.  Instead  the  system  was  bounded  by 
the  knowledge  necessary  to  analyze  the  error  data  and  create  maintenance  suggestions  for  a  certain 
class  of  switch  problems.  In  retrospect,  the  developers  of  the  system  feel  that  more  attention  should 
have  been  paid  to  requirements  and  to  testing. 

A  single  expert  was  utilized  to  provide  domain  knowledge.  The  guidelines  used  to  select  the  domain 
expert  were:  must  be  recognized  as  an  expert  by  his  peers  in  other  GTE  telephone  companies, 
must  he  articulate,  must  be  enthusiastic  about  the  project,  and  must  be  available  for  knowledge 
acquisition  one  week  per  month  for  at  least  three  years.  The  knowledge  engineers  together  with 
the  expert  created  KAR  documents.  The  knowledge  acquisition  process  naturally  ended  when  a 
particular  switch  problem  was  properly  diagnosed  and  appropriate  maintenance  suggestions  were 
provided  for  the  test  data  under  consideration.  The  development  tool  KEE  was  used  to  create  the 
knowledge  base  which  consisted  of  frames  and  production  rules. 

Rapid  prototyping  was  not  used  very  much  in  the  development  of  COMPASS.  Some  of  the  devel¬ 
opers  feel  that  this  was  a  definite  shortcoming.  To  track  updated  versions  of  the  system,  a  software 
control  knowledge  base  was  used.  This  controlled  how  the  system  was  to  be  built  from  the  various 
files  and  also  how  changes  and  additions  were  to  be  saved.  Actual  end  users  were  not  involved  in 
the  development  process.  End  users  were  represented  by  the  domain  expert  who  was  a  supervisor 
of  pol  enl.ial  end  users 

COMPASS  is  comprised  of  the  following  components:  a  language  processor  which  was  supplied  by 
KEE,  and  knowledge  bases  -  some  containing  facts  only  and  some  containing  mostly  rules.  There 
is  no  justifier,  interpreter,  scheduler,  or  consistency  enforcer  in  the  system.  Communications  is  not 
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part  of  the  system  but  is  handled  by  conventional  software  on  conventional  hardware  for  speed  and 
efficiency. 

Depending  on  the  type  of  diagnostic  task,  COMPASS  contains  up  to  eighteen  knowledge  bases 
with  over  1000  frames  and  15000  slots.  Five  of  the  knowledge  bases  contain  over  500  production 
rules  and  there  are  more  than  500  LISP  procedures  in  the  system.  The  user  interface  is  handled 
by  a  single  knowledge  base  with  20  frames  and  approximately  50  LISP  procedures.  The  support 
environment  is  handled  by  one  knowledge  base  with  10  frames  and  20  procedures. 

To  evaluate  the  COMPASS  system,  independent  experts  were  asked  to  read  the  KAR  documents 
and  to  assess  the  system’s  output  given  some  input.  Small  test  files  containing  error  messages  for 
one  or  more  problems  were  created  and  used  to  validate  the  system.  The  test  files  emphasized  rule- 
path  coverage  through  the  various  COMPASS  phases.  Boundary  value  and  stress  testing  was  also 
done.  This  was  accomplished  using  test  files  which  contained  a  small  number  of  messages  and  single 
problems.  Regression  testing  was  emphasized  after  modifications  were  made.  Documentation  of 
the  testing  phase  was  limited  to  describing  general  procedures  and  overall  results.  LISP  procedures 
were  written  to  perform  the  regressing  testing  in  batch  mode.  Criteria  for  a  test  to  be  considered  a 
success  was:  for  a  given  input,  did  the  appropriate  rules  fire  and  was  the  proper  output  produced? 
Appropriate  and  proper  were  determined  from  the  KAR  documents.  Another  performance  issue 
was  the  ability  of  the  COMPASS  system  to  perform  with  the  same  accuracy  as  experts. 

The  product  developers  of  the  COMPASS  system  feel  that  some  improvements  in  the  design  pro¬ 
cedures  should  be  made.  There  should  have  been  more  consultation  with  the  potential  end  users 
of  the  system  and  a  life  cycle  should  have  been  defined.  The  product  developers  should  have  been 
involved  earlier  in  the  life  cycle,  rather  than  just  the  prototype  developers.  A  high  level  design 
document  should  have  been  produced  early  in  the  life  cycle  and  more  attention  should  have  been 
paid  to  traditional  software  engineering  techniques  (e.g.  maintainability  is  more  important  than 
elegance  and  efficiency). 


C.10  IBM  Federal  Systems  Group  Summary 

IBM  Federal  Systems  Group  reported  on  their  Fault  Diagnosis  and  Resolution  System  (FDRS), 
which  is  a  hardware  failure  diagnosis  for  the  ground-based  equipment  of  the  Air  Force’s  satellite 
command  and  control  facility.  The  system  diagnoses  the  failure  down  to  the  level  for  which  there 
is  a  redundant  component  and  recommends  the  proper  replacement  procedures. 

FDRS  was  developed  with  independent  research  and  development  funds  by  4  engineers,  one  of  whom 
had  previous  AI  experience.  An  initial  problem  definition  phase  was  completed  in  approximately 
3  man-months,  and  the  first  prototype  was  developed  in  an  additional  3  man-months.  The  entire 
project  took  approximately  20  man-months  to  develop.  The  final  prototype  was  tested  in  with 
the  actual  command  and  control  software  using  a  hardware  simulation,  but  currently  has  not  been 
embedded  in  the  operational  system. 

FDRS  satisfies  the  criterion  of  demonstrating  the  feasibility  of  a  new  technology  Productivity 
gains  and  user  satisfaction  have  not  been  determined  yet  since  the  system  is  not  installed  in  the 
operational  environment.  The  system  has  satisfactorily  met  two  additional  criteria;  they  are:  speed 
of  execution,  and  integration  into  the  main  system. 
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The  development  approach  used  to  build  the  FDRS  system  consisted  of  the  following  steps: 

•  Identification 

•  Conceptualization 

•  Formalization 

•  Implementation 

•  Testing 

The  above  steps  were  repeated  to  provide  more  depth  of  knowledge  with  each  prototype.  Docu¬ 
mentation  which  was  produced  included  a  detailed  description  of  the  knowledge,  design  and  code 
documentation.  Informal  design  reviews  and  code  inspections  were  held. 

No  requirements  analysis  was  performed.  Instead,  the  problem  was  initially  bound  by  a  general 
statement  of  wanting  to  prove  the  feasability  of  a  knowledge-based  approach  for  this  problem.  The 
second  prototype  was  analyzed  to  determine  what  additional  requirements  would  be  included  in 
the  final  version. 

Knowledge  was  not  acquired  through  a  domain  expert,  rather  the  engineers  studied  the  requirements 
document  for  fault  diagnosis  and  recovery.  Tables  were  constructed  for  each  device  indicating  each 
possible  fault,  the  symptoms  of  a  failure,  tests  which  should  be  performed,  and  recovery  actions. 
The  knowledge  was  represented  as  production  rules. 

The  tools  used  in  the  development  process  varied  with  each  prototype.  For  the  first  prototype  a 
commercially  available  shell  was  used.  The  knowledge  based  system  in  the  second  prototype  was 
developed  using  a  small  internal  knowledge  based  system  shell.  This  prototype  interfaced  with 
a  Pascal  simulation  which  was  1/3  of  the  code.  The  final  prototype  was  JOVIAL  code  which 
was  integrated  into  the  command  and  control  system.  Each  new  prototype  was  considered  a 
separate  research  effort  and  underwent  the  standard  research  approval  process.  Each  developer 
was  responsible  for  certain  areas  of  the  software.  This  individual  was  the  only  one  authorized  to 
make  updates  to  his  assigned  area.  All  software  followed  a  naming  convention  to  track  the  current 
versions  of  the  system. 

The  FDRS  system  consists  of  a  knowledge  base  and  an  interpreter.  A  uniform  knowledge  represen¬ 
tation  was  used,  as  well  as  a  separate,  simple  inference  engine.  In  terms  of  size,  the  knowledge  base 
contains  approximately  100  rules.  The  following  is  a  list  of  the  components  and  the  percentage  of 
the  total  system. 


Component 


Percentage 


Knowledge  Base 

20% 

Inference  Engine 

50% 

User  Interface 

20% 

Support  Environment 

10% 
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No  formal  test  plans  were  developed,  instead  each  path  of  the  knowledge  base  was  tested  by  going 
back  to  the  original  table  of  possible  failures  and  testing  to  see  if  each  failure  path  was  followed 
in  the  expected  manner.  Unit  tests  were  performed  on  each  separate  rule  base,  and  integration 
testing  wa3  performed  on  those  rule  bases  which  interacted  with  each  other.  The  FDRS  system 
was  tested  within  a  simulation  of  the  command  control  system. 

C.ll  Inference  Corporation  Summary  (Authorizer’s  Assistant) 

Inference  Corporation  reported  on  an  Authorizer’s  Assistant  system  which  was  built  for  Ameri¬ 
can  Express,  Inc.  The  system  aids  human  authorizers  in  evaluating  complicated  charge  rases  to 
determine  if  the  charge  should  be  authorized. 

The  Authorizer’s  Assistant  was  developed  by  8  people.  The  project  personnel  was  broken  down  as 
follows:  1  Manager,  4  Knowledge  Engineers,  2  System  Engineers,  and  1  Technical  Writer,  all  with 
previous  Al  experience.  The  first  prototype  was  developed  in  15  man-months,  and  72  man-months 
later  the  fielded  system  was  completed. 

In  terms  of  demonstrating  the  feasibility  of  a  new  technology  and  overall  user  satisfaction,  the 
system  is  considered  a  success.  Productivity  gains  are  in  the  process  of  being  measured.  Inference 
noted  additional  criteria  for  this  system  as  the  direct  savings  due  to  fraud  loss  reduction,  and  the 
less  tangible  improvement  in  customer  relations  which  are  promoted  by  the  systems  explanation 
facility. 

The  following  are  the  steps  utilized  in  Inference’s  development  approach. 

•  Knowledge  Acquisition 

•  Design 

•  Implementation 

•  Knowledge  Engineering/Implementation 

•  System  Interfaces 

•  Validation  Sc  Development  Tools 

•  Commercial  Systems  Coding 

•  Expertise  Validation  Sc  Refinement 

•  Acceptance 

The  Knowledge  Acquisition  and  Design  steps  were  performed  iteratively  to  produce  a  prototype 
(Implementation).  With  the  prototype  complete,  the  steps  of  Knowledge  Engineering,  Implementation. 
Systems  Interfaces,  and  Validation  and  Development  Tools  were  performed  and  produced  a  com¬ 
plete  system.  From  the  complete  system  the  steps  of  Commercial  Systems  Coding  and  Expertise 
Validation  and  Refinement  were  performed.  When  these  steps  were  complete,  the  project  was  ready 
for  the  final  step  of  Acceptance.  Throughout  this  process,  there  were  discussions  and  reviews  with 
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the  customer.  Documentation  that  resulted  included:  a  Design  Document,  Project  Organisation 
Charts,  Schedules,  Users  Manual,  and  a  Manager’s  Manual. 

The  requirements  analysis  phase  performed  was  focused  on  the  activity  of  knowledge  engineering. 
Users  were  included  as  experts  and  a  Design  Document  was  produced.  The  process  was  bounded 
by  continuous  discussions  and  reviews  with  the  customer. 

The  method  used  to  acquire  knowledge  for  the  Authorizer’s  Assistant  was  as  follows: 

1.  Watch  autliorizers  work 

2.  Team  review  of  cases  off  line  with  experts 

3.  On  going  process  of  Knowledge  Base  refinement 

4.  Review  of  code  by  experts 

5.  Review  of  system’s  behavior  of  real  cases  off  line  with  experts 

6.  Doing  the  job  and  learning  first  hand 

The  knowledge  acquired  was  represented  using  production  rules.  The  system’s  knowledge  base  has 
expanded  and  the  decision  not  to  include  additional  knowledge  was  a  managerial  one  determined 
by  the  constraints  of  the  contract.  Six  experts  were  used,  and  they  were  chosen  by  the  customer’s 
internal  review  process  which  is  based  on  service,  productivity  and  accuracy  of  work.  One  expert 
was  used  predominately  more,  who  settled  inconsistencies  and  discrepancies. 

Inference’s  ART  development  tool  was  used,  along  with  Symbolics  Common  LISP.  Users  were 
involved  throughout  the  development  process.  The  prototype  was  a  subset  of  the  final  system,  it 
was  not  and  never  intended  to  be  “throw  away  code”.  Design  changes  did  occur  over  the  course 
of  development,  but  the  fundamental  design  did  not  change.  Approval  for  the  changes  was  via  the 
project  manager  and  interaction  with  the  customer,  when  needed.  The  system  is  connected  to  a 
IBM  transaction  system.  This  proved  to  be  a  very  time  consuming  process.  There  was  no  formal 
software  configuration  management  on  the  project. 

The  Authorizer’s  Assistant  has  a  separate  inference  engine,  a  uniform  knowledge  representation, 
language  processor,  incremental  compiler  as  an  interpreter,  a  communications  handler,  and  a  sym¬ 
bolic  scheduler.  The  knowledge  base  consists  of  800  rules.  The  following  is  the  division  among  the 
types  of  code: 

•  Knowledge  Base  20% 

•  ART  Inference  Engine  40% 

•  User  Interface  12% 

•  Environment  28% 

No  formal  test,  procedures  were  developed.  Code  was  written  to  partially  automate  the  review 
process  such  that  sets  of  test  cases  could  be  run  in  batch  mode.  Users  were  involved  extensively 
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in  the  testing  process.  The  expert  system  results  were  compared  with  the  experts  conclusions  and 
found  to  be  correct  97%  of  the  time.  Testing  was  done  at  the  system  level  only  The  system  is  not 
self  modifying. 

System  performance  was  very  important.  Performance  was  measured  along  three  axes.  CPU, 
swapping  and  garbage.  The  code  was  “instrumented”  to  find  out  where  CPU  was  being  used  and 
modifications  were  then  made  to  various  parts  of  the  system.  Inference  feels  that  the  performance 
improvements  were  made  too  late  in  the  development  process,  it  would  have  been  easier  and  faster 
to  have  made  design  decisions  all  along  that  would  have  contributed  to  a  high  performance  system 
User  suggestions  were  made,  and  negotiation  was  used  to  determine  if  they  would  be  implemented 

The  following  is  a  list  of  Inference’s  lessons  learned  from  this  project. 


1.  More  design  should  be  done  up  front. 

2.  Performance  monitoring  should  be  done  throughout  the  development  process. 

3.  More  test  beds  should  have  been  developed  for  simulating  various  parts  of  the  customer's 
systems  to  avoid  the  dependency  on  last  minute  testing. 

4.  Application  of  more  “classic”  software  engineering  techniques. 

5.  Use  of  management  tools  for  actively  tracking  the  project:  (i.e.  Gantt  and  pert  charts  and 
automated  schedulers). 


C.12  Inference  Corporation  Summary  (Medical  Charge  Evalu¬ 
ation  Control) 

Inference  Corporation  reported  on  a  Medical  Charge  Evaluation  and  Control  (Medchec)  system 
which  evaluates  medical  claims  and  prioritizes  them  with  regard  to  possible  mischarging.  Inference 
has  had  extensive  experience  building  Knowledge  Based  systems;  prior  to  developing  this  system 
they  had  built  somewhere  in  the  vicinity  of  10  -  20  systems.  Medchec  is  being  developed  under 
contract  by  3  engineers,  one  with  previous  Al  experience.  The  lirst  prototype  was  built  in  6 
man  mont  hs 

In  terms  of  overall  user  satisfaction  the  Medchec  system  is  so  far  considered  a  success.  Inference  feels 
this  to  be  the  most  important  criteria  for  success.  As  far  as  productivity  gains  and  demonstration 
of  a  new  technology  the  system  is  also  considered  successful.  Inference  noted  that  another  criteria 
against  which  to  measure  success  was  the  degree  to  which  the  developed  approach  can  be  easily 
extrapolated  to  other  applications. 

The  following  are  the  steps  and  products  used  in  Inference’s  iterative  development  approach. 
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Products 


Identification 

Conceptualization 

Formalization 

Implementation 

Testing 


Proposal 

Rule  set  description 


Data  structures 


Code/ prototype 
Test  case  panel  results 


It  is  important  to  note  that  these  steps  were  not  performed  once  in  a  sequential  manner.  Rather, 
all  steps  except  identification  could  be  entered  from  the  previous  and/or  succeeding  steps.  Re¬ 
views  were  held  at  approximately  1  month  intervals.  These  reviews  included  high  level  overviews, 
demonstrations,  reports  produced  from  the  expert  system,  and  PERT  charts  of  current  progress. 

A  requirements  analysis  phase  was  performed  in  which  the  users,  who  were  also  the  experts,  par¬ 
ticipated.  The  experts  were  interviewed  and  asked  how  they  would  audit  hundreds  of  claims  if  they 
had  the  time.  This  system  does  riot  mimic  any  current  operations  but  instead  performs  a  depth  of 
analysis  never  done  before  The  users/experts  completely  defined  the  requirements  of  the  system. 

Knowledge  was  acquired  from  3  experts  during  group  discussions  in  which  very  little  conflict  arose. 
The  knowledge  was  grouped  as  follows. 

1  experience  of  past  mischarging 

2  expectations  of  mischarging  patterns 
3.  suggestions  by  knowledge  engineer 

The  information  was  documented  in  the  form  of  taxonomy  and  English-like  “rules”.  The  knowledge 
was  represented  in  frames  and  rules.  The  frames  were  used  to  index  demons  which  computed 
patterns  of  repetition  of  a  service  and  costs  of  a  service.  The  frames  were  also  used  to  linearize  and 
combine  the  various  assertions  of  interesting  patterns  for  reporting  purposes.  The  rules  were  used 
to  detect  each  pattern  of  mischarging,  one  main  rule  per  pattern.  The  knowledge  base  expanded 
as  “recommended  by  the  experts”. 

Development  tools  used  were  Inference’s  Automated  Reasoning  Tool  (ART)  and  LISP  on  a  Sym¬ 
bolics  3675  Rapid  prototyping  was  also  employed.  The  philosophy  on  the  Medchec  project  was  to 
build  prototypes  as  a  subset  or  framework  of  the  completed  system.  From  the  start  it  was  designed 
to  be  extensible  and  expandable  both  in  performance  and  capability  of  supporting  all  types  of 
knowledge  and  inferencing  or  reasoning  techniques.  The  prototypes  were  not  throw  away  systems. 
Many  low  level  design  changes  occured  with  no  approval  necessary  to  implement  changes.  Also, 
there  was  no  formal  software  configuration  management  on  the  project. 

The  Modeller  system  has  a  separate  inference  engine,  a  uniform  knowledge  representation  of  facts 
and  rules,  a  communication  handler,  and  a  justifier  which  was  used  to  flush  obsoleted  facts.  The 
system  size  is  small  and  uses  forward  and  backward  chaining  reasoning.  The  initial  knowledge  base 
consists  of  50  rules  The  communication  handler  consists  of  3000  to  4000  lines  of  LISP  to  handle 
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C.13  Lockheed  Aircraft  Service  Company  Suumiary  (Expert  Software  Pricer) 


data  parsing  from  the  claims  database  on  a  S-mini  computer.  Also  included  in  the  system  is  about 
140  schemata  for  loss  types  and  diagnoses.  The  following  is  the  division  among  the  types  of  code 

•  Knowledge  base  30% 

•  Inference  Engine  10%  -  enhancement  of  ART  features 

•  User  interface  20% 

•  Support  environment  40% 

No  formal  development  plans  were  developed  to  test  the  system.  The  experts  review  the  reports 
generated  by  the  system  to  assess  accuracy.  The  Medchec  system  incorporates  feedback  from 
auditors  on  confirmed  mischarging  to  alter  its  ratings. 


C.13  Lockheed  Aircraft  Service  Company  Summary  (Expert 
Software  Pricer) 

Lockheed  Aircraft  Service  Company  (LAS)  reported  on  an  Expert  Software  Pricer  (ESP)  system 
for  software  costing.  Included  in  the  ESP  system  is  a  knowledge  based  Expert  Sizer  which  assists 
in  estimating  the  size  of  the  software  system  being  bid.  Once  determined,  the  size  can  be  input 
into  one  of  ESP’s  pricing  models. 

ESP  was  an  internally  funded  project  and  the  first  A1  project  developed  by  LAS.  A  two  person  team 
developed  the  first  prototype  in  4  man-months.  It  took  12  man-months  to  complete  the  system. 
The  schedule  was  defined  by  the  funding  that  was  allocated.  Where  applicable,  LAS  Software 
Engineering  Procedures  were  followed. 

LAS  feels  that  the  system  is  successful  in  terms  of  demonstrating  the  feasibility  of  a  new  technology 
and  achieving  significant  productivity  gains.  The  ESP  system  has  demonstrated  the  capability  of 
applying  AI  techniques  to  software  costing  while  also  providing  savings  in  software  bidding  time 
and  effort.  In  terms  of  user  satisfaction,  LAS  is  encouraging  the  use  of  the  system  within  the 
Lockheed  Corporation. 

The  development  approach  used  to  build  the  ESP  system  consisted  of  the  following  stages: 

•  Identification 

•  Conceptualization 

•  Formalization 

•  Implementation 

•  Integration 

•  Demonstrate/Test 
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C.13  Lockheed  Aircraft  Service  Company  Sunmiary  (Expert  Software  Pricer) 


About  2  internal  reviews  per  month  were  held  during  the  development  process.  Tint**  reviews 
involved  status  reporting  and  demonstrations.  A  requirements  analysis  phase  was  performed  which 
yielded  a  Software  Requirements  Specification  document. 

Knowledge  was  acquired  from  examining  source  code  and  documentation  from  completed  software 
systems.  The  functions  of  the  systems  and  their  associated  sizes  were  then  documented.  Knowledge 
acquisition  is  considered  to  be  an  ongoing  process.  The  knowledge  acquired  was  represented  as 
frames  with  incomplete  knowledge  denoted  by  flagged  dummy  information. 

The  ESP  system  was  developed  using  the  Lockheed  Expert  System  (LES)  shell  LES  includes  a 
backward  chaining,  goal  driven  inference  engine.  The  user  interface  included  in  LES  was  enhanced 
to  make  it  more  user  friendly  and  accommodate  the  highly  interactive  nature  of  the  Expert  Sizer. 
In  an  attempt  to  limit  the  solution  space,  each  type  of  software  system  to  be  sized  (i.e.  avionics 
systems)  is  represented  as  a  separate  knowledge  base.  Rapid  prototyping  was  performed  very  early 
in  the  design  conceptualization  phase.  Users  helped  identify  bugs  and  made  recommendations  for 
improvement  during  the  development  process. 

ESP  is  under  configuration  management  using  the  VAX/VMS  CMS  system.  A  Software  Change 
Request  must  be  approved  before  any  change  can  be  made.  Once  the  change  is  completed,  a 
Software  Change  Description  must  be  generated  before  the  change  is  incorporated. 

The  LES  shell  provided  the  following  elements: 


•  Language  Processor 

•  Justificr 

•  Interpreter 

•  Scheduler 

•  Consistency  Enforcer 

•  Communications  Handler 


The  knowledge  base  consists  of  77  rules  with  700  lines  of  factual  knowledge.  The  user  interface  and 
support  environment  written  for  ESP  consists  of  1200  lines  of  Pascal  code. 

There  were  no  formal  test  procedures  used  to  validate  the  ESP  system.  Systems  with  known  size 
and  costs  were  used  as  test  cases.  If  the  ESP  estimates  were  within  +/-  80%  of  the  actuals,  then 
the  system  results  were  considered  acceptable. 

Knowledge  acquisition  has  been  a  difficult  and  time-consuming  task.  It  is  felt  that  a  tool  to  allow 
users  lo  input  their  site  specific  sizing  data  is  needed  both  to  alleviate  the  system  developers  from 
the  knowledge  acquisition  task  and  to  allow  users  to  more  easily  customize  the  knowledge  base. 
LAS  plans  to  build  such  a  tool  this  year. 
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C.14  Lockheed  Aircraft  Service  Company  Summary  (Frequency 
Hopper  Signal  Identifier) 


Lockheed  Aircraft  Service  Company  (LAS)  reported  on  a  Frequency  Hopper  Signal  Identifier  sys¬ 
tem  which  detects  and  characterizes  frequency  hopped  signals.  The  A1  component  aids  in  signal 
identification.  This  was  an  internally  funded  project  and  the  second  A1  project  developed  by  LAS. 
One  person,  with  prior  A1  experience,  developed  the  first  prototype  in  9  months. 

The  system  is  considered  a  success  in  terms  of  demonstrating  feasibility  of  a  new  technology.  As 
far  as  significant  productivity  gains,  the  system  has  been  successful  for  some  but  not  all  possible 
signals/signal  environments. 

A  requirements  analysis  phase  was  performed  which  yielded  a  technical  proposal  and  report  de¬ 
scribing  the  theory  and  implementation  in  detail.  User  involvement  in  this  process  was  considered 
very  beneficial. 

Knowledge  was  acquired  through  a  literature  review  and  interviews  with  experts.  The  experts 
provided  a  small  amount  of  very  useful  information.  The  knowledge  acquisition  is  considered  to 
be  an  ongoing  activity.  The  knowledge  acquired  was  represented  in  a  temporal  framework  with 
confidence  levels  attached  to  most  inferences. 

The  system  was  developed  in  Common  LISP  on  a  micro  Vax.  Iterative  cycles  of  design  and  imple¬ 
mentation  were  employed.  The  magnitude  of  changes  was  large  with  no  prior  approval  necessary. 
Rapid  prototyping  was  used  to  test  feasibility  of  implementation  and  to  identify  shortcomings  in 
design.  The  iterative  cycle  consisted  of  the  following  steps: 

•  Implementation 

•  Refine  prototype 

•  Test  to  determine  shortcomings 

•  Upgrade  and  expand  prototype 

The  prototype  was  never  frozen.  User  participation  greatly  influenced  the  development  process 

The  architecture  of  the  system  consists  of  a  uniform  knowledge  representation  and  a  simple  inference 
engine.  The  system  contains  the  following  components: 


•  Knowledge  base 

•  .luslilier 

•  Scheduler 

•  Consistency  enforcer 

The  size  of  the  components  listed  below  is  in  terms  of  the  number  of  LISP  functions: 


•  Knowledge  base  -  50 


CM  5  Lockheed- Georgia  Company  Summary 


•  Inference  engine  -  6t' 

•  Lser  interface  -  '20 

•  Support  environment  -  40 

No  formal  test  procedures  were  developed.  The  testing  strategies  consisted  of  simulated  real  time 
response;  i.e.  how  fast  could  a  hopper  signal  be  detected  and  characterized,  also,  how  do  a  wide 
variety  of  noisy  signal  environments  effect  performance.  The  criteria  for  system  tests  was  that  of 
consistency  and  improved  performance  rather  than  pass/ fail  - 

The  system  developer  feels  that  taking  the  time  to  try  to  obtain  a  general  knowledge  of  the  ap¬ 
plication.  and  how  conventional  methods  approach  the  problems  can  be  counterproductive  since  it 
tends  to  channel  thinking  along  conventional  lines.  This  was  also  true  to  some  extent  in  reviewing 
AI  approaches. 


C.15  Lockheed-Georgia  Company  Summary 

The  Lockheed-Georgia  Company  (LGC),  under  a  research  and  development  contract  from  the  U.S. 
Air  Force,  is  developing  a  Pilot’s  Associate  system.  The  objective  of  the  system  is  to  provide 
pilots  of  single  seat  fighter  aircraft  a  near  real  time  on  board  support  system.  The  system’s  jobs 
include  monitoring  the  mission  environment,  evaluating  each  situation,  and  providing  intelligence 
to  the  pilot  on  the  current  capabilities  of  his  aircraft  and  the  tactics  deemed  usable  in  a  specific 
situation.  The  information  provided  to  the  pilot  is  analyzed  against  the  mission  or  alternate  mission 
objectives. 

The  project  team  consists  of  over  40  engineers.  65%  of  the  team  has  had  some  previous  experience  in 
Al  and/or  Expert  Systems.  All  of  the  engineers  on  the  project  have  had  prior  experience  developing 
software  for  the  Department  of  Defense. 

The  Pilot’s  Associate  is  currently  in  the  analysis  stage.  It  is  expected  to  take  over  240  man-months 
to  conduct  the  analysis,  develop  a  simulation  package,  conduct  two  demonstrations,  and  deliver 
documentation  to  the  customer  under  Phase  I.  Significant  productivity  gains  are  scheduled  for 
Phase  II,  since  knowledge  base  designs  will  have  been  established.  Phase  II  completion  is  scheduled 
for  February,  1989  and  aims  at  developing  real  time  processing  of  cooperative  Expert  Systems.  The 
success  of  this  project  will  be  measured  by  two  demonstrations  of  a  portion  of  the  overall  task. 

System  development,  of  the  Pilot’s  Associate  is  based  on  the  rapid  prototyping  of  elements  within 
the  system  components  and  integration  builds  around  the  mission  manager  executive  which  is  a 
central  system  blackboard.  Informal  reviews  are  held  prior  to  each  integration  build.  In  addition, 
formal  design  reviews  are  scheduled  during  the  three  year  project. 

The  requirements  analysis  phase  was  completed  thirteen  months  after  the  start  of  the  contract. 
This  phase  included  rapid  prototyping  to  define  requirements  for  each  system  component.  System 
Specifications  and  Subsystem  Description  documents  were  produced  for  each  component  of  the 
system 

The  knowledge  acquisition  in  this  project  is  accomplished  via  documented  interviews  with  Air  Force 
fighter  pilots.  These  interviews  are  reviewed  by  a  technical  review  board  consisting  of  Lockheed, 


C.16  MITRE/Bedford  Summary 


Air  Force,  and  contracted  experts.  Knowledge  is  also  acquired  from  tin*  following  sources:  lliglii 
engineers,  A/C  design  specialists,  manuals  and  Air  Force  studies.  All  facts  shared  hot  ween  the 
developers  are  posted  on  an  electronic  bulletin  board  for  review  by  other  experts.  All  documen¬ 
tation  is  approved  prior  to  incorporation  into  the  technical  database  of  the  contract.  Approval  is 
required  prior  to  each  knowledge  base  change  and  is  granted  at  the  project  management  level  Users 
are  included  in  the  following  aspects  of  the  development  process:  reviews,  knowledge  acquisition, 
demonstrations  of  the  prototype  systems,  and  quarterly  scheduling 

The  development  tools  employed  on  the  Pilot’s  Associate  project  are:  ART,  LES,  and  OPS5.  All 
subsystem  developers  have  a  major  commercial  tool  or  a  mature  internally  developed  tool.  An 
audit  trail  is  being  used  to  track  design  decisions/design  rational.  The  design  changes  are  approved 
by  program  management.  The  prototype  releases  are  under  contractor  configuration  control  and 
are  subject  to  control  review  prior  to  release  for  integration.  Baselines  are  being  used  to  “freeze 
in  time”  portions  of  the  system.  These  baselines  are  under  contractor  control.  The  configuration 
audit  trail  is  maintained  by  using  a  Code  Management  System  on  a  DEC  VAX  System 

The  Pilot’s  Associate  project  has  several  expert  systems  integrated  into  the  product  The  system 
architecture  is  characterized  by  the  following  items: 
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•  Knowledge  bases  and  inference  engines  are  primarily  separate 

•  Several  knowledge  bases  must  be  “translated”  to  be  understood  by  one  another 

•  Most  inference  engines  are  simple 


•>r- 

.>  ,%.-S 


•  Overlapping  knowledge. 

All  subsystems  contain  a  knowledge  base,  scheduler,  and  communications  handler.  A  language 
processor  and  justifier  are  also  components  of  the  system  The  mission  manager  handles  consistency 
enforcement. 


For  system  evaluation/validation  the  '  ilot’s  Associate  has  developed  and  documented  test  plans 
A  Test  Directorate,  made  up  of  experts  and  software  engineers,  develops  test  cases  for  formal 
review.  The  tests  are  conducted  step  by  step  and  witnessed  by  personnel  independent  of  the 
design  personnel.  A  documented  record  is  maintained  for  each  test.  Testing  is  rated  as  passed 
if  the  system  requirements  are  satisfied.  Experts  will  evaluate  system  performance  against  the 
specified  performance  of  objectives  and  selected  test  cases.  Currently  no  tools  are  being  user!  in  the 
testing  phase.  Quarterly  demonstrat  ions  and  reviews  are  conducted  by  management  and  customer  - 
personnel. 


C.16  MITRE/Bedford  Summary 

MITRE  reported  on  the  Liquid  Oxygen  Expert  System  which  was  built  for  NASA  Kennedy  Space 
Center.  The  purpose  of  the  system  is  fault  detection  and  diagnosis  of  bad  sensors  and  components 
in  the  liquid  oxygen  loading  component  of  the  Launch  Processing  System. 


t  s 

A 

£ 


MICROCOPY  RESOLUTION  TEST  CHART 
■  JRf  All  vUNn»«n<:  i%vi 


C.l  7  MITRE/McLean  Summary 


The  project  development  team  consisted  of  2  people:  one  MITRE  engineer  with  6  years  experience 
in  building  AI  systems  and  one  NASA  engineer  with  no  Ai  experience  but  considered  the  domain 
expert.  Neither  had  experience  with  military  software  standards. 

The  Liquid  Oxygen  Expert  System  project  was  funded  on  a  yearly  basis  and  there  was  no  prede¬ 
fined  schedule.  However,  the  first  prototype  was  completed  in  6  months  and,  after  2  years,  the 
final  prototype  is  being  used  in  background  mode.  The  system  is  considered  a  success  in  that  it 
demonstrated  the  feasibility  of  a  new  technology  and  achieved  overall  user  satisfaction. 

The  first  3  months  of  the  project  focused  on  the  problem  definition  in  which  the  users  were  in- 
dispensible  equal  partners.  Requirements  were  never  frozen,  but  rather  evolved  as  did  the  team’s 
understanding  of  the  problem. 

The  knowledge  acquisition  process,  which  is  never  considered  complete,  involved  informal  discus¬ 
sions  with  NASA  personnel,  review  of  existing  software  and  documentation  that  described  the 
existing  software.  The  knowledge  representation  method  used  was  frame  based  since  it  best  re¬ 
flected  the  structure  of  the  domain.  Throughout  the  development  of  the  system,  the  knowledge 
base  did  expand  with  the  approval  of  the  domain  expert.  Programming  tools  used  were  Symbolic’s 
ZETALISP  and  FRL  from  MITRE  and  MIT. 

During  the  development  process  of  the  final  prototype,  several  “sub-prototypes”  were  built.  Better 
ideas  spurred  major  design  changes  which  were  then  implemented  as  the  basis  of  a  new  prototype. 
Neither  designs  nor  prototypes  were  ever  frozen;  changes  were  made  upon  consensus  of  the  domain 
expert.  Because  the  domain  expert  is  a  user  of  the  system,  there  was  considerable  user  input 
throughout  the  development  period. 

The  principal  components  of  Liquid  Oxygen  Expert  System  are  the  knowledge  base,  a  consistency 
checker,  a  fault  location  algorithm  and  an  interface  to  the  user  or  outside  sensors.  The  components 
were  chosen  for  appropriateness  and  decided  by  team  consent.  In  terms  of  conventional  software, 
the  system  interfaced  to  Modcomp  minicomputers  over  a  communications  link  for  sensor  data 
transmission. 

The  system  was  tested  as  it  was  developed,  against  test  cases  and  live  sensor  data  -  its  performance 
is  judged  by  whether  or  not  the  users  agree  with  its  conclusions. 

Overall,  close  user/domain  expert/developer  interaction  had  a  strong  and  beneficial  effect  on  sys¬ 
tem  development.  The  absence  of  a  formal  development  procedure  was  an  extremely  positive 
environmental  factor,  leading  to  a  strong  problem  solving  environment  and  high  morale. 

C.17  MITRE/McLean  Summary 

MITRE/McLean  reported  on  the  ANALYST  system  which  was  deployed  to  the  9th  Infantry  Di¬ 
vision  in  Ft.  Lewis,  Washington.  Further  plans  call  for  using  the  system  as  a  test  bed  at  both 
Ft.  Sill  and  Ft.  Leavenworth  for  DARPA  projects  through  fiscal  year  1988.  ANALYST  grew  out 
of  sponsored  research  to  analyze  and  apply  production  system  techniques  to  military  applications. 
'I'he  effort  evolved  into  a  contract  to  develop  a  general  know  ledge- based  aid  for  intelligence  use. 
'Phe  system  processes  sensor  returns  from  different  intelligence  sources  and  displays  a  transient 
situation  of  enemy  combat  units  in  real-time. 
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C.18  Northrop/ Aircraft  Division  Summary 


The  project  development  team  consisted  of  five  people,  including  one  domain  expert,  and  one 
managing  A1  scientist.  The  estimated  development  time  for  the  first  prototy|w  which  did  not 
emphasise  performance  was  six  to  eight  months.  The  following  year  and  a  half  was  devoted  to 
enhancing  system  performance  capability  and  building  a  knowledge  base  to  facilitate  the  user 
modifications. 

About  eighteen  months  were  spent  doing  requirements  analysis.  During  this  time,  an  A  minus 
specification  for  ANALYST  was  produced.  Requirements  Analysis  paralleled  rapid  prototyping 
and  thus  influenced  the  choice  of  AI  technology  and  tools. 

ANALYST  was  developed  using  in-house  and  LISP  tools.  The  only  tool  purchased  was  a  micro¬ 
compiler  for  the  LISP  machine.  There  were  procedural  differences  between  the  development  of 
ANALYST  as  an  internal  project  and  development  for  the  field  prototype.  There  were  a  number  of 
observations  made  during  development.  It  was  recognised  that  the  lack  of  standards  was  compen¬ 
sated  for  by  the  use  of  AI  and  LISP  machines.  Configuration  control  become  more  defined  when 
the  project  started  focusing  on  a  delivery  date.  Control  extended  to  files  within  a  delivery  build. 
Changes  in  code  were  documented  with  comments  and  verified  by  the  programmer  who  made  the 
modification.  During  development,  the  system  was  also  used  as  a  test  bed  for  spatial  and  temporal 
techniques  as  well  as  evidential  reasoning. 

During  the  testing  phase,  a  test  of  the  inference  engine  and  access  to  the  knowledge  and  data  bases 
was  performed.  A  critical  issue  with  the  ANALYST  system  was  the  ability  to  interrupt  during  a  test 
for  a  what-if  question  in  order  to  explore  the  consequences  of  a  particular  path.  The  method  tested 
for  absences  of  firm  requirements  and  complete  specifications.  Since  several  experts  with  different 
opinions  were  available,  testing  become  even  more  difficult.  However,  the  user  community  was 
conservative  in  their  changes  and  considered  the  effects  on  the  inference  engine  before  suggesting  a 
change.  Regression  tests  were  also  performed  to  ensure  that  previous  prototype  capabilities  were 
maintained  intact. 

Overall,  the  need  for  a  prototype  with  user  involvement  in  the  demonstration  or  lest  phase  was 
considered  important.  Once  mature,  it  was  stated  that  the  prototype  should  be  released  for  some 
limited  operational  testing  after  a  much  shortened  in-plant  acceptance  test.  Hardware  should  be 
subjected  to  the  standard  testing  cycle.  The  need  for  a  specification  requirement  for  an  explana¬ 
tion  of  the  system’s  capability  was  recognised.  The  concept  of  training  a  domain  expert  about 
knowledge  engineering  was  favored  over  teaching  the  knowledge  engineer  about  the  domain  since 
the  latter  is  a  time-consuming  factor.  However,  the  knowledge  engineer  was  considered  to  be  the 
main  implementor  and  director.  In  addition,  the  net'll  for  software  discipline  and  standards  was 
emphasized  particularly  with  respect  to  the  need  for  a  specifications  document  as  well  as  design 
and  control  principles. 

C.18  Northrop/ Aircraft  Division  Summary 

Northrop/Aircraft  Division  reported  on  the  Expert  System  for  Target  Attack  Sequencing  (ESTAS). 
ESTAS,  an  internally  funded  system,  is  a  real-time  system  that  provides  decision  aids  for  pilots 
under  stressful  and  high-workload  missions  segment,  i.e.,  the  combat  phase.  Some  of  the  decision 
aids  are  navigation  verification,  target  prioritisation,  target  selection,  weapon  allocation  to  targets, 
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C.1S  Northrop/ Aircraft  Division  Summary 


and  weapon  readiness  preparation.  The  system  is  expected  to  provide  responses  under  all  conditions 
including  unpredictable  ones. 

ESTAS  is  designed  to  be  one  component  of  a  highly  integrated,  flight  simulation  system.  Its  re¬ 
sponse  is  very  dependent  on  the  action/behavior  of  other  components.  At  this  time,  however, 
schedule  and  funding  constraints  precluded  the  integration  of  ESTAS  with  the  overall  flight  simu¬ 
lation  system. 

The  ESTAS  software  development  team  consisted  of  four  engineers,  two  with  previous  Al  experi¬ 
ence.  In  addition,  two  of  the  four  engineers  were  experienced  in  conventional  software  development 
for  DOD.  The  development  of  the  system  from  problem  definition  to  (irst  prototype  took  Nix  montliN 
or  two  man-years.  Management  predefined  the  schedule,  and  no  follow-on  effort  to  Held  the  system 
followed. 


ESTAS  was  developed  within  the  framework  of  the  traditional  software  development  cycle  with 
the  exception  that  iteration  amongst  phases  was  acceptable.  The  specific  model  used  to  develop 
ESTAS  consisted  of  the  following  phases: 


analysis  -  document  the  operational  environment  which  included  mission  profiles  and  timelines; 


definition  -  document  the  system  functional  requirements  and  applicable  areas  for  AI  software 
development; 


identification  -  document  specific  input  and  output  criterion,  and  general  description  of  the 
application  requirements  based  on  problem  characteristics; 


conceptualization  -  report  on  AI  concepts  and  techniques  that  satisfy  application  requirements; 


knowledge  acquisition  -  document  rules  and  facts; 


formalization  -  expert  system  shell  code  and  related  documents; 


implementation  -  updated  document  of  rules  and  facts  as  well  as  code  the  knowledge  base; 
and 


testing  -  test  result  reports. 


Iterations  through  formalization  and  testing  were  an  inherent  aspect  of  the  software  development 
effort.  Requirements  were  frozen  at  the  end  of  the  analysis  phase  to  avoid  modification  problems. 
Prototyping  began  after  the  conceptualization  phase.  Documentation  was  enforced  but  not  based 
on  the  MIL-STD  documentation  practices.  The  documentation  included  software  requirements, 
top-level  design  and  detailed  design.  Technical  reviews  to  assess  system  progress,  as  well  as  analyse 
and  solve  problems  were  held  periodically.  Progress  reports  were  given  to  the  developers.  System 
demonstrations  at  the  end  of  formalization,  implementation  and  testing  were  also  provided. 


Knowledge  acquisition  consisted  of  domain  research  by  the  knowledge  engineer  and  interrogations 
of  the  expert  which  included  questionnaires,  round-table  discussions/interviews,  and  acting  out 
of  hypothetical  situations.  Laboratory  simulators  were  used  to  illustrate  the  experts  views  and 
comments.  The  means  used  to  gather  the  expert  information  involved  note  taking  and  tapes. 
Then,  both  were  transformed  into  prose  and  eventually  pseudo-code  which  underwent  a  formal 
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C.l 9  PAR  Government  Systems  Corporation  Summary 


approval  proem.  Knowledge  acquisition  was  considered  complete  when  the  expert  approved  the 
knowledge  base  for  the  specific  case.  At  this  time,  the  knowledge  base  consists  of  171  rules.  The 
inference  mechanism  used  is  based  on  both  forward  and  backward  chaining. 

The  rule-based  knowledge  representation  method  was  selected  because  of  the  need  for  real-time 
system  responses.  At  this  time  there  are  no  means  for  accomodating  ill-defined  or  incomplete 
knowledge.  The  chosen  representation  was  reviewed  during  a  design  walkthrough.  The  knowledge 
base  continued  to  expand  through  informal  reviews  and  a  formal  approval  process  until  the  first 
prototype  release.  The  only  tool  used,  the  LISP  workstation,  incorporated  required  tools  such  as 
a  language  (Common  Lisp),  a  graphics  implementation  and  a  developer  interface. 

The  major  components  of  the  system  are  a  knowledge  base,  a  separate  inference  engine  and  a  user 
interface.  Cost  and  schedule  constraints  precluded  the  implementation  of  a  language  processor,  a 
justifier  and  a  consistency  enforcer.  The  architecture  underwent  a  formal  review  and  approval  by 
technical  associates  and  the  project  engineer. 

Performance  of  ESTAS  was  primarily  based  on  user  satisfaction  and  the  expert’s  evaluation  follow¬ 
ing  incremental  additions  to  the  knowledge  base  or  inference  engine.  Discrete  components  were  not 
tested  separately.  Other  than  tracing  facilities  built  into  the  system,  tools  were  not  used  for  testing. 
Because  ESTAS  was  not  integrated  into  the  overall  system,  rigorous  testing  was  not  possible  nor 
desirable. 

Several  observations  were  made  throughout  the  development  of  ESTAS.  First,  system  decomposi¬ 
tion  was  believed  critical  to  the  successful  integration  of  an  AI  system  to  a  larger  system.  Second,  it 
is  believed  that  Al  developments  must  go  through  the  traditional  system  development  cycle  (  iter¬ 
ations  acceptable)  involving  the  required  engineering  disciplines.  The  implication  is  that  engineers 
across  the  various  disciplines  must  be  knowledgeable  in  AI.  Third,  the  expert  must  be  assigned 
and  committed  to  the  project.  Expert  participation  on  an  “as  available”  basis  hindered  progress 
on  ESTAS. 


C.19  PAR  Government  Systems  Corporation  Summary 

PAR  Government  Systems  Corporation  (PGSC)  reported  on  three  systems:  Duplex  Army  Ra¬ 
dio/Radar  Targeting  Decision  Aid  (DART),  Cost  Benefit  of  Tactical  Air  Operations  (CBTAO),  and 
See  and  Project  Enemy  Activity  (SPEA).  They  are  all  decision  aids  for  the  Tactical  Air  Control 
Center,  developed  to  the  point  of  a  working  prototype.  None  of  the  systems  have  been  deployed. 

The  DART  Aid  is  designed  to  assist  the  Command,  Control,  and  Communications  Countermeasures 
(C'C’M)  Analyst  in  the  identification/classification  of  the  following  targets: 

•  Unidentified  Command  Post  (UICP) 

•  Air  Defense  Regimental  Headquarters 

•  SA-8  Battery 

•  SA-6  Battery 
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C.19  PAR  Government  Systems  Corporation  Summary 


•  SA-4  Battery /Battalion/ Brigade 

•  Division  Main  Headquarters 

•  Division  Forward  Command  Post 

•  Division  Alternate  Headquarters 

•  Radio  Relay  Stations 

•  ZSU  23-4 

•  EW  Radars 

The  CBTAO  decision  aid  allocates  tactical  air  resources  into  mission  packages,  based  on  the  highest 
probability  of  success.  The  system  also  provides  the  planners  with  estimates  and  explanations  of 
the  cost  and  benefit  of  such  tactical  mission  packages.  The  planners  using  the  system  can  accept 
or  modify  the  recommended  force  packages  based  on  his  judgements. 

The  SPEA  decision  aid  assists  the  tactical  air  resources  allocation  and  employment  process  by 
providing  planners  with  the  ability  to  systematically  project  the  dispositions  of  enemy  forces  up 
to  72  hours  in  advance.  The  force  projections  may  be  adjusted  as  necessary  to  compensate  for 
perceived  differences  between  the  SPEA  projections  and  the  real  world.  The  end  product  is  a 
“planning  picture”  to  be  used  by  Combat  Plans  in  interpreting  the  situation  for  the  next  day’s  Air 
Tactical  Operations  (ATO). 

All  three  decision  aids  were  contractually  funded.  DARI’  was  developed  under  the  Decision  Aids 
for  Target  Aggregation  (DATA)  RADC  contract.  Both  CBTAO  and  SPEA  were  developed  under 
the  Integrated  Tactical  Air  Control  Center  (ITACC)  RADC  contract. 

At  PC  SC,  one  A I  schooled  engineer  is  tasked  to  work  on  a  given  decision  aid.  The  Ai  engineer  is 
team  leader  where  a  team  can  be  composed  of  3  to  5  computer  scientists  and  one  or  more  in-house 
domain  experts.  Essentially  all  the  engineers  have  prior  experience  in  developing  software  for  the 
government. 

The  development  process  used  to  build  the  decision  aids  consisted  of  the  following  phases: 

•  Task  I  -  Analysis  and  Design  :  preparation  of  Problem  Definition  Statement  (PDS)  and  a 
high  level  design  structure  of  the  system.  Includes  requirements  analysis  wherein  the  domain 
experts  define  the  problem; 

•  Task  II  -  Development  and  Documentation  :  building  a  prototype  or  vertical  slice  of  the 
system,  and  preparing  technical,  test  and  user  documentation; 

•  Task  III  -  lest  and  Evaluation  :  system  evaluation,  from  a  technical  and  operational  stand¬ 
point 


The  knowledge  acquisition  process  was  accomplished  via  sessions  with  knowledge  engineers  and 
government  supplied  experts.  Knowledge  gained  was  recorded  either  by  tape  or  by  hand.  In-house 
experts  or  consultants  were  then  used  to  verify  the  accuracy  of  the  information  provided.  In  cases 


C.20  Sanders  Associates,  Jnc.  Summary 


of  questionable  knowledge,  a  consensus  approach  was  used.  At  PGSC,  the  preference  is  to  train  an 
in-house  domain  expert  to  be  a  knowledge  engineer  to  work  with  the  government  experts  rather 
than  use  a  knowledge  engineer  with  little  or  no  domain  experience.  For  the  three  decision  aids,  the 
knowledge  was  implemented  in  the  form  of  production  rules  with  confidence  factors. 

PGSC  has  internally  developed  several  expert  system  shells  that  use  a  rule  based,  probabilistic 
inferencing  mechanism.  Each  decision  aid  was  developed  using  one  of  the  in-house  shells. 

For  each  system,  PGSC  built  a  working  prototype  which,  when  coupled  with  the  PDS,  defined 
the  final  product  and  provided  a  proof  in  principle  for  the  finished  system.  Users  and  experts 
were  introduced  to  the  system  at  the  prototype  stage,  approximately  8  to  12  months  into  system 
development. 

Both  DART  and  CBTAO  contain  a  knowledge  base  (facts/rules),  justifier,  rule  compiler,  inference 
engine,  and  scheduler.  In  the  DART  system  the  knowledge  base  contains  222  rules  and  20,709  total 
lines  of  code.  The  DART  system  was  developed  using  an  in-house  shell  written  iti  Pascal.  The 
graphics  and  data  base  interfaces  were  written  in  C. 

In  the  CBTAO  system  the  knowledge  base  contains  50  rules  and  12,886  total  lines  of  code.  The 
CBTAO  aid  was  developed  on  the  Symbolics  3600  and  a  Tektronix  4125  was  used  for  color  graphics 
display.  A  GKS  package  was  developed  to  provide  an  interface  to  the  Tektronix. 

The  SPEA  system  was  built  with  object  oriented  programming  using  the  Flavors  package  on  the 
Symbolics  3600.  The  SPEA  system  is  6,000  total  lines  of  code  and  contains  115  Flavors  Definitions. 

Formal  Test/Evaluation  Plans  were  developed  for  all  three  decision  aids.  The  systems  were  tested 
as  entities  both  technically  and  operationally.  Technical  evaluation  was  performed  by  a  group  of  ex¬ 
perts  and  nonexperts  with  computer  science/  engineering  backgrounds.  Operational  evaluation  was 
performed  by  potential  users  who  responded  to  questionnaires.  The  questionnaires  were  designed 
to  extract  the  users’  perception  of  the  systems  in  terms  of  strengths,  weaknesses  and  suggested 
improvements. 


C.20  Sanders  Associates,  Inc.  Summary 

Sanders  Associates  Inc.  reported  on  an  internally  funded  effort  to  build  an  intelligent  assistant 
for  the  task  of  reprogramming  automatic  test  equipment.  The  purpose  of  TESS,  the  test  assistant 
program,  was  to  develop  a  technology  base  and  capture  some  specific  knowledge  about  testing  of 
ECM  systems. 

There  were  three  participants  from  the  Al  side  of  the  house;  experience  was  minimal,  although  two 
had  successfully  pursued  academic  studies  in  AI.  The  domain  expert  was  from  the  user  community. 
Substantially  less  than  half  time  was  available  from  the  domain  expert/uscr  representative 

TESS  personnel  agreed  with  the  steps  outlined  in  the  proposed  development  cycle.  They  stressed 
the  iterative  nature  of  the  identification-conceptualization-formalization-implementalion  subcycle. 
This  was  driven  predominantly  by  the  need  to  generalize  specific  cases  after  enough  problem  un¬ 
derstanding  was  achieved. 
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C.21  SAScE  Summary  (Decision  Support  System ) 


Products  of  the  various  stages  included  an  IRAD  plan  in  the  identification  phase,  constraint  denti¬ 
tions  and  user  interfaces  in  the  formalization  phase,  and  knowledge,  data  layout  language  and 
attached  primitives  in  the  implementation  phase. 

Knowledge  reorganization  occurred  within  the  FRL  (Frame  Representation  Language)  to  accomo¬ 
date  generalization;  sometimes  this  reorganization  would  require  some  functional  extension  in  the 
FRL.  The  changes  came  from  increased  understanding  of  required  functionality.  The  only  controls 
on  changes  were  the  need  for  team  agreement.  The  largest  changes  were  in  the  area  of  the  user 
interface  and  the  knowledge  base  restructuring.  It  was  felt  that  these  areas  represented  something 
like  exploratory  programming. 

Reviews  were  informal  day  long  working  sessions  with  AI  project  personnel,  and  a  consultant.  The 
main  emphasis  was  on  direction  of  future  efforts  with  little  time  spent  on  reviewing  past  work. 

The  requirements  analysis  phase  was  used  to  set  goals  and  directions.  Many  details  of  the  require¬ 
ments  were  deferred  until  prototyping  was  finished.  No  documentation  was  produced  explicitly 
from  the  requirements  analysis  phase.  There  were  IRAD  plans,  notes  from  review  meetings,  and 
the  immediate  code  generation  (i.e.  documentation  of  system  as  built/executable).  Requirements 
were  not  frozen  but  were  left  in  abeyance.  Users  were  included  in  the  requirements  analysis  to  the 
extent  their  project  loading  could  spare  them. 

Testing  was  informal  and  done  on  small  chunks  of  code.  There  was  some  reliance  on  repetitive 
execution  to  provide  confidence  in  the  reliability  of  the  modules. 

Tool  usage  was  centered  around  the  development  system  afforded  by  the  Symbolics  3600  system. 
Some  additional  tools  that  came  out  of  the  TESS  experience  were  FRL,  the  Data  Layout  Language 
(DLL),  and  the  constraint  propagation  language  (CPL). 

Frames  were  selected  as  the  basis  of  knowledge  representation  based  on  a  consultants  review  of  the 
project  and  its  features.  There  was  some  support  for  incomplete  or  uncertain  knowledge  including 
decision  postponement. 

The  TESS  developers  were  not  able  to  judge  the  completeness  of  the  experts  knowledge.  They  felt 
that  there  was  not  enough  involvement  by  the  expert.  Several  items  that  they  would  look  for  in 
an  expert  are:  availability,  commitment,  interest,  recent  and  on-going  practice,  competence,  com¬ 
munication  skills.  Areas  of  interface  and  control  with  the  expert  were  characterized  by  needing  to 
establish  a  tasking  or  contractural  relationship;  periodic  performance  reviews,  and  shared  physical 
accomodations. 

C.21  SA&E  Summary  (Decision  Support  System) 

SAAE  reported  on  a  decision  support  system  which  is  a  classified  project.  The  prototype  was 
completed  within  the  first  year.  Subsequent  releases  are  produced  every  three  months.  The  system’s 
development  time  has  been  reduced  because  of  a  tool  built  under  the  same  contract,  the  Decision 
Support  Development  System  (l)SDS),  which  aids  in  the  creation  of  run  time  systems. 

The  SAAK  software  development  teams  consist  of  two  engineers  to  develop  the  decision  support 
system  and  fifteen  engineers  assigned  to  DSDS.  One  of  the  engineers  had  extensive  Al  experience. 
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C.22  SAScE  Summary  (Sensitive  Financial  Analysis  System) 


Also,  one  engineer  had  minimal  experience  with  the  Department  of  Defense  software  development 

process. 

The  general  process  followed  in  the  development  of  the  decision  support  system  consisted  of  the 
following  phases:  requirements  definition,  specification,  validation,  generation,  and  verification. 
The  requirements  analysis  performed  involved  a  definition  of  goals,  the  impact  of  design  options  on 
schedule  and  the  system  architecture  and  an  iterative  redefinition  of  requirements  for  each  incre¬ 
mental  system  release.  Documentation  consisted  of  a  requirements  specification  that  the  customer 
and  development  manager  reviewed  and  signed  ofT  for  each  increment,  a  design  document  for  each 
iteration,  and  informal  knowledge  acquisition  memos  distributed  to  customers  and  the  design  team. 

Knowledge  acquisition  involves  regular  interviews  with  the  experts/users  and  additional  research 
on  decision  support.  Apparently,  the  incremental  approach  to  software  development  helps.  Experts 
are  given  a  number  of  opportunities  to  supply  knowledge  information  and  to  define  the  rules  which 
govern  the  decision  support  system. 

The  knowledge  representation  method  is  rule-based.  The  DSDS  environmental  development  tool 
used  by  the  decision  support  system,  makes  use  of  Knowledge  Engineering  System  (KES),  Unify, 
LISP  procedures  and  GE  Scan. 

The  major  components  of  the  system  are  a  knowledge  base,  a  justifier,  an  inference  engine,  a 
language  processor  (LISP),  a  scheduler,  an  interpreter,  and  a  communications  handler  (TCP/IP). 

Alpha  and  beta  tests  are  performed  on  the  SA&E  system.  Alpha  tests  ensure  that  the  integrated 
system  works.  It  is  then  given  to  the  application  engineers  for  rigorous  testing.  Beta  lest  involves 
a  user  demonstration  of  the  system  for  evaluation  and  comments.  Assessment  of  user  satisfac¬ 
tion  is  accomplished  via  video  records  of  customer  reactions  to  the  product.  Acceptable  system 
performance  is  determined  by  the  experts. 

Overall,  the  development  and  availability  of  DSDS  vastly  improved  the  decision  support  system’s 
development  time.  Through  the  use  of  the  DSDS  environmental  tool,  a  large  production  expert 
system  can  be  more  easily  developed  and  maintained. 


C.22  SA&E  Summary  (Sensitive  Financial  Analysis  System) 

SAAE  reported  on  a  sensitive  system  which  was  developed  for  an  unnamed  client.  The  system, 
which  was  contractually  funded,  analyzes  financial  data.  The  prototype  was  developed  within  three 
months  followed  by  the  final  product  eight  months  later. 

The  SAJkE  software  development  team  included  two  engineers,  of  which  one  had  previous  AI 
software  development  experience.  Neither  engineer  had  previous  experience  with  conventional 
software  development  for  the  Department  of  Defense. 

The  developers  found  the  following  software  development  model  appropriate  to  the  AI  project: 


•  Identification  -  determining  problem  characteristics 

•  Categorization  -  categorizing  the  domain  knowledge 


C.23  7 exas  Instrumenta  Inc.  Summary 


•  Structuring  -  form  the  framework  for  the  knowledge  by  count  meting  the  attribute  hierarchy 

•  Implementation  -  capture  knowledge 

•  Testing  -  validate  the  knowledge  base  and  system  behavior 

A  requirements  analysis  was  performed.  The  system  was  built  incrementally.  Each  increment 
consisted  of  a  functional  system  with  more  capabilities.  Informal  reviews  and  approvals  followed. 
No  documentation  was  required  by  the  customer.  However,  informal  notes  were  distributed. 

Knowledge  acquisition  involved  the  following  process.  First,  specific  statements  were  defined  and 
used  as  a  premise  for  judgements  the  system  was  expected  to  be  capable  of  performing.  Second, 
the  data  needed  to  produce  the  judgement  was  identified.  Third,  the  logic  required  to  make 
the  connection  between  data  and  judgements  was  determined.  Finally,  case  studies,  with  known 
outcomes  were  used  to  identify  incomplete  knowledge. 

The  knowledge  representation  method  was  rule-based.  Tools  used  were  the  Knowledge  Engineering 
System  (KES),  which  is  a  high-level  expert  system  shell,  a  standard  text  editor  and  the  KES  parser. 

The  major  components  of  the  system  are  a  knowledge  base,  a  justifier,  a  consistency  enforcer,  a 
communications  handler  and  an  inference  engine.  A  selection  of  available  software  tools  marketed 
by  SA&E  determined  the  system’s  architecture. 

The  system  was  evaluated  by  running  a  series  of  case  studies  with  known  outcomes  through  the 
system.  Experts  decided  whether  or  not  the  system  met  the  performance  requirements.  Both  the 
system  and  the  individual  rules  were  tested. 

Overall,  the  regular  interactions  between  the  experts  and  the  engineers  led  to  the  success  of  the 
system.  Experts  were  allowed  an  opportunity  to  familiarize  themselves  with  the  system,  determine 
what  kind  of  data  the  developing  engineers  needed,  and  to  correct  and  expand  the  system. 


C.23  Texas  Instruments  Inc.  Summary 

Texas  Instruments  Inc.  reported  on  the  Production  Scheduler  Expert  System  which  was  to  provide 
a  scheduling  mechanism  for  textile  fiber  production  and  inventory.  The  system  should  have  been 
commercially  funded  prior  to  each  of  the  following  three  phases:  the  demo  system  phase;  the 
prototype  system  phase;  and  the  final  production  system  phase.  However,  funding  was  removed 
three  months  before  the  prototype  release  due  to  a  lack  of  customer  resources.  The  estimated 
completion  time  for  the  final  product  phase  was  two  years. 

The  Production  Scheduler  software  development  team  consisted  of  one  knowledge  engineer  to  man¬ 
age  and  implement  the  system  and  two  domain  experts  that  defined  the  knowledge  and  constraints. 
The  knowledge  engineer  had  a  good  working  knowledge  of  AI  tools  and  of  Lisp  programming 
techniques  and  conventional  software  development,  but  little  Al  software  development  experience. 
Other  management  and  engineering  resources  were  available.  Minimal  reviews  between  the  knowl¬ 
edge  engineer  and  the  domain  experts  transpired.  The  meetings,  although  limited,  were  found 
extremely  valuable. 
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C.23  Texas  Instruments  In c.  Summary 


No  formal  requirements  analysis  was  performed.  The  requirement  problems  identified  resulted  from 
a  previous  attempt  to  resolve  the  problem  through  conventional  programming  techniques. 

The  general  process  used  in  the  development  of  the  Production  Scheduler  to  dale  consisted  of  the 
following  phases: 

•  Identification  -  determining  the  feasibility  of  the  application  based  on  the  limited  scope  of  the 
problem  and  the  ability  to  gather  a  static  collection  of  facts  and  constraints. 

•  Conceptualization  -  a  frame  based  knowledge  representation  was  chosen.  Also,  an  initial 
knowledge  network  was  defined. 

•  Formalization  -  designing  structures  to  organize  knowledge. 

•  Implementation  -  gathering  knowledge  via  the  user. 

The  structure  of  the  knowledge  base  was  modified  several  times  throughout  implementation.  As 
previously  mentioned,  no  prototype  was  released.  Formal  documentation  was  not  provided  at  the 
end  of  the  identification,  conceptualization  or  formalization  phases. 

Minimal  reviews  between  the  knowledge  engineer  and  the  domain  expert  were  allowed  The  lack 
of  direct  access  to  the  domain  expert  and  the  customer’s  management  hindered  the  successful 
acquisition  of  knowledge.  Also,  incorrect  assumptions  were  made  by  the  knowledge  engineer  which 
resulted  in  disillusioned  users.  When  the  project  was  terminated,  knowledge  acquisition  was  not 
complete. 

The  knowledge  representation  method  is  frame  based.  The  entire  system  development  met  with 
no  formal  approval  process.  Tools  used  to  develop  the  software  were  LISP  and  a  windowing  forms 
management  tool. 

The  major  components  of  the  system  were  an  interpreter,  a  scheduler  and  a  user  interface.  In 
addition,  a  communication  handler  had  beeh  planned. 

A  formal  evaluation  process,  although  scheduled,  did  not  transpire  since  the  system  never  reached 
completion.  At  the  implementation  phase,  the  domain  expert  checked  the  schedule  test  date  for 
accuracy.  The  interface  was  tested  separately.  Then,  both  were  tested  together. 

Overall,  the  customer’s  availability  and  committment  as  well  as  sufficient  funds  to  complete  the 
project  were  determined  as  necessary,  but  lacking  for  the  Production  Scheduler  project  The  rela¬ 
tionship  between  the  knowledge  engineer  and  the  domain  expert  cultivates  the  user's  view  of  the 
system.  Without  regular  reviews,  the  user's  expectations  of  the  system  may  not  be  satisfied  The 
ability  to  acquire  knowledge  from  user’s,  and  the  customer’s  management  should  be  encouraged 
Also,  acceptable  system  behavior  should  be  documented. 
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Rome  Air  Development  Center  <| 

RAUC  plans  and  executes  fie.6e.axch,  development,  test 
and  selected  acquisition  pxogxams  in  suppoxt  oi 
Command,  Contxol,  Communications  and  Intelligence 
( C -5 1 )  activities.  Technical  and  engineexing 
suppoxt  within  axeas  o^  competence  is  pxovided  to 
ESV  Pxogxam  O^ices  [POs)  and  othex  ESV  elements 
to  pexfoxm  elective  acquisition  o&  C3l  systems. 

The  axeas  o&  technical  competence  include 
communications ,  command  and  contxol,  battle 
management,  in^oxmation  pxocessing,  Suxveillance 
sensoxs,  intelligence  data  collection  and  handling, 
solid  state  sciences ,  electxo magnetics ,  and 
pxopagation,  and  etectxonic,  maintainability , 
and  compatibility. 
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